Debian 8 package list error

carcosa

I have a debian 8 vps that has been running for some time now, today i tried to add nixstats agent to it for monitoring and got error messages, seems they are coming from running apt-get update?

root@code:~# apt-get update
Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:2 http://security.debian.org jessie/updates/main Sources [202 kB]
Get:3 http://security.debian.org jessie/updates/main amd64 Packages [478 kB]
Get:4 http://security.debian.org jessie/updates/main Translation-en [238 kB]
Ign http://deb.debian.org jessie InRelease
Get:5 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:6 http://deb.debian.org jessie Release.gpg [2434 B]
Get:7 http://deb.debian.org jessie-updates/main Sources [17.2 kB]
Get:8 http://deb.debian.org jessie-updates/main Translation-en [14.9 kB]
Get:9 http://deb.debian.org jessie Release [148 kB]
Get:10 http://deb.debian.org jessie-updates/main amd64 Packages [20.1 kB]
Get:11 http://deb.debian.org jessie/main Sources [7054 kB]
Get:12 http://deb.debian.org jessie/main amd64 Packages [6789 kB]
Get:13 http://deb.debian.org jessie/main Translation-en [4583 kB]
Fetched 19.8 MB in 4s (4291 kB/s)
Reading package lists... Error!
E: Unable to synchronize mmap - msync (5: Input/output error)
E: The package lists or status file could not be parsed or opened.

I have 50% disk space on the server and have gone as far as to re create the sources list for jessie and still get this error?

sources.list

deb  http://deb.debian.org/debian jessie main
deb-src  http://deb.debian.org/debian jessie main
deb  http://deb.debian.org/debian jessie-updates main
deb-src  http://deb.debian.org/debian jessie-updates main
deb http://security.debian.org/ jessie/updates main
deb-src http://security.debian.org/ jessie/updates main

Googling suggests this may be related to bad sectors on the drive, has anyone ever run into this issue before?

WSS

run free -h && df -h

carcosa

output:

root@code:~# free -h && df -h
             total       used       free     shared    buffers     cached
Mem:          2.0G       362M       1.6G       5.4M        14M       223M
-/+ buffers/cache:       124M       1.8G
Swap:         1.0G         0B       1.0G
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1      1007G  512G  445G  54% /
udev             10M     0   10M   0% /dev
tmpfs           403M  5.4M  397M   2% /run
tmpfs          1006M  8.0K 1006M   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs          1006M     0 1006M   0% /sys/fs/cgroup

WSS

Well, you're not out of memory, and your HD isn't full, so that's good.

Try running:

apt-get -s clean

Decide if you agree with what it's going to do, and if so,

apt-get clean
then
apt-get update && dpkg -f install

then, you might consider a reboot into single user mode, and run fsck -A if that isn't resolved.

---

@Falzo said:
is this KVM or OVZ? if the former maybe this hints to a thin-provisioned sparse image and no space left on the node to grow... just a wild guess.

I've never seen a /dev/vda on OVZ...

Falzo

@WSS said:
run free -h && df -h

also try to run apt-get clean

is this KVM or OVZ? if the former maybe this hints to a thin-provisioned sparse image and no space left on the node to grow... just a wild guess.

carcosa

It is a KVM from ZXhost at Hetzner.........

i tried apt-get clean with same results

root@code:~# apt-get update && dpkg -f install

Produces the same result also?

WSS

Are you running torrents or something similar? You don't show out of disk space, but it sure acts like you have no space available. I'd reboot the system in single user mode and run fsck.

mksh

Try dmesg | tail -n20 to see if there is anything about a disk failure. Also you can safely delete all files under /var/lib/apt/lists as apt-get update will just recreate them. Might be worth a try to do mount -t tmpfs tmpfs /var/lib/apt/lists && mkdir /var/lib/apt/lists/partial. That should rule out any kind of disk problem.

carcosa

@mksh

root@code:~# dmesg | tail -n20
[ 3714.268009] end_request: I/O error, dev vda, sector 1153197952
[ 3714.268009] EXT4-fs warning (device vda1): ext4_end_bio:317: I/O error -5 writing to inode 39378481 (offset 0 size 8388608 starting block 144149870)
[ 3714.298677] JBD2: Detected IO errors while flushing file data on vda1-8
[ 3719.824213] JBD2: Detected IO errors while flushing file data on vda1-8
[ 4041.003339] blk_update_request: 181 callbacks suppressed
[ 4041.003350] end_request: I/O error, dev vda, sector 1153240064
[ 4041.003786] EXT4-fs warning: 181 callbacks suppressed
[ 4041.003799] EXT4-fs warning (device vda1): ext4_end_bio:317: I/O error -5 writing to inode 26 (offset 0 size 413696 starting block 144155109)
[ 4041.003802] buffer_io_error: 22770 callbacks suppressed
[ 4041.003804] Buffer I/O error on device vda1, logical block 144154752
[ 4041.004166] Buffer I/O error on device vda1, logical block 144154753
[ 4041.004569] Buffer I/O error on device vda1, logical block 144154754
[ 4041.004946] Buffer I/O error on device vda1, logical block 144154755
[ 4041.005347] Buffer I/O error on device vda1, logical block 144154756
[ 4041.005817] Buffer I/O error on device vda1, logical block 144154757
[ 4041.006189] Buffer I/O error on device vda1, logical block 144154758
[ 4041.006587] Buffer I/O error on device vda1, logical block 144154759
[ 4041.006976] Buffer I/O error on device vda1, logical block 144154760
[ 4041.007329] Buffer I/O error on device vda1, logical block 144154761
[ 4784.164546] perf interrupt took too long (10043 > 10000), lowering kernel.perf_event_max_sample_rate to 12500

Running:

mount -t tmpfs tmpfs /var/lib/apt/lists && mkdir /var/lib/apt/lists/partial

Results in the same error.

Starting a backup now

mksh

@carcosa said:
@mksh

root@code:~# dmesg | tail -n20
[ 3714.268009] end_request: I/O error, dev vda, sector 1153197952
[ 3714.268009] EXT4-fs warning (device vda1): ext4_end_bio:317: I/O error -5 writing to inode 39378481 (offset 0 size 8388608 starting block 144149870)
[ 3714.298677] JBD2: Detected IO errors while flushing file data on vda1-8
[ 3719.824213] JBD2: Detected IO errors while flushing file data on vda1-8
[ 4041.003339] blk_update_request: 181 callbacks suppressed
[ 4041.003350] end_request: I/O error, dev vda, sector 1153240064
[ 4041.003786] EXT4-fs warning: 181 callbacks suppressed
[ 4041.003799] EXT4-fs warning (device vda1): ext4_end_bio:317: I/O error -5 writing to inode 26 (offset 0 size 413696 starting block 144155109)
[ 4041.003802] buffer_io_error: 22770 callbacks suppressed
[ 4041.003804] Buffer I/O error on device vda1, logical block 144154752
[ 4041.004166] Buffer I/O error on device vda1, logical block 144154753
[ 4041.004569] Buffer I/O error on device vda1, logical block 144154754
[ 4041.004946] Buffer I/O error on device vda1, logical block 144154755
[ 4041.005347] Buffer I/O error on device vda1, logical block 144154756
[ 4041.005817] Buffer I/O error on device vda1, logical block 144154757
[ 4041.006189] Buffer I/O error on device vda1, logical block 144154758
[ 4041.006587] Buffer I/O error on device vda1, logical block 144154759
[ 4041.006976] Buffer I/O error on device vda1, logical block 144154760
[ 4041.007329] Buffer I/O error on device vda1, logical block 144154761
[ 4784.164546] perf interrupt took too long (10043 > 10000), lowering kernel.perf_event_max_sample_rate to 12500

Ok, that kinda explains it.

Running:

mount -t tmpfs tmpfs /var/lib/apt/lists && mkdir /var/lib/apt/lists/partial

Thats strange though as package lists should be written to RAM by mounting /var/lib/apt/lists on tmpfs. Guess there is other places where it wants to read/write also.

Starting a backup now

Probably a good idea as the disk doesn't seem all that healthy.

huntercop

Have you tried reporting the issue to the host to investigate and resolve?

carcosa

Yes first thing i did but its ZXhost and as of late not confident ill get a reply or a solution with how everything has been handled, so i asked here in the meantime.

WSS

Well, this is the first time I can recall that SpinRite has been the right answer.