Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 145803 reports (Page 15 of 2917)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X............X.......X.............................] duration: 549.345735ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:45:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.992317ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.............X.....................X......................] duration: 499.276822ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:45:46 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 499.299711ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.................X...........................] duration: 695.040047ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:45:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 695.069445ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........X.................................] duration: 647.119363ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:45:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 647.143816ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X.......X........................] duration: 500.731751ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:45:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 500.760121ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....X.......................] duration: 420.104479ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:45:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 420.195053ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.................X..........................] duration: 522.201684ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:45:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 522.230073ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X............X.......X............................] duration: 472.780623ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:45:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.808538ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.................X.........................] duration: 478.893915ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 478.923679ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.............X.....................X.....................] duration: 494.014093ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:44:35 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 494.038218ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X............X.......X...........................] duration: 506.98406ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:44:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.072978ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.................X........................] duration: 493.62679ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 493.65279ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.......X.......................] duration: 713.93948ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:44:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 713.980237ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....X......................] duration: 383.73494ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:44:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 383.765948ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X..........X................................] duration: 597.60093ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:44:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 597.626807ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.................X.......................] duration: 1.604236926s trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:43:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 1.604265021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X............X.......X..........................] duration: 482.654652ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:43:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 482.684631ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.............X.....................X....................] duration: 442.018752ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:43:39 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 442.046607ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.................X......................] duration: 477.554694ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 477.580533ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X..........X...............................] duration: 582.854629ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:43:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.879241ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.......X......................] duration: 492.614045ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:43:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 492.686162ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X............X.......X.........................] duration: 568.900283ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:43:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 568.917742ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.................X.....................] duration: 473.134574ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:43:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 473.157537ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X..........X..............................] duration: 591.973383ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:43:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 592.015135ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....X.....................] duration: 491.180543ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:42:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 491.208435ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.............X.....................X...................] duration: 428.434979ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:42:55 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 428.473948ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.................X....................] duration: 507.905005ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:42:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 507.934092ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X.......X.....................] duration: 456.445752ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.525604ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.............X.....................X..................] duration: 572.347824ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:42:07 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 572.371414ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.................X...................] duration: 511.957527ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:42:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 511.985221ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X..........X.............................] duration: 565.697028ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:42:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 566.2802ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.....X....................] duration: 438.730932ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:42:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 438.759886ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X............X.......X........................] duration: 452.688907ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:42:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.736626ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X.......X....................] duration: 478.836948ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:41:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 478.902674ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.................X..................] duration: 535.653627ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:41:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 535.680401ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.................X.................] duration: 474.898558ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:41:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 474.93062ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.....X...................] duration: 563.646879ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:41:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 563.677039ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X..........X............................] duration: 607.491359ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:41:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 607.991854ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.............X.....................X.................] duration: 463.240251ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:40:55 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 463.269747ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X.......X...................] duration: 470.859691ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:40:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.884404ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X............X.......X.......................] duration: 448.867408ms trace: https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91 FAIL (2025-10-31 21:40:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.890374ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..........X...........................] duration: 579.728348ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:40:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 579.76983ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.................X................] duration: 493.156516ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:40:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 493.182535ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.....X..................] duration: 387.157107ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 387.189469ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X.......X..................] duration: 537.437471ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:40:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 537.468192ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.................X...............] duration: 431.108244ms trace: https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55 FAIL (2025-10-31 21:40:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded ======LOGS====== 825937748s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece... Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments... Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944 , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286 , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111 , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850 , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823 , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043 , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683 , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956 , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579 , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829 , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996 , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846 , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832 , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091 , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil> Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785 , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675 , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540 , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625 , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments... Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil> Duration: 431.136508ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.............X.....................X................] duration: 407.589459ms trace: https://jaeger.dev.storj.io/trace/2a65f5f21f735664 FAIL (2025-10-31 21:39:56 UTC) operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments... Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment... Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment. Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece... Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments... Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006 , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731 , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799 , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798 , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840 , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837 , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439 , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618 , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546 , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874 , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223 , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil> Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698 , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807 , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770 , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939 , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806 , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments... Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 407.617216ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.....X.................] duration: 443.854466ms trace: https://jaeger.dev.storj.io/trace/22de3cd654292a7d FAIL (2025-10-31 21:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments... Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment... Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment. Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece... Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments... Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894 , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504 , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225 , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990 , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188 , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859 , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267 , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576 , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil> Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277 , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000 , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016 , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856 , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093 , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652 , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605 , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078 , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151 , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235 , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments... Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 443.881222ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X.......X.................] duration: 533.700595ms trace: https://jaeger.dev.storj.io/trace/689f32b5a76d000d FAIL (2025-10-31 21:39:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 533.720339ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X..........X..........................] duration: 568.116299ms trace: https://jaeger.dev.storj.io/trace/464ff9dd561aefd2 FAIL (2025-10-31 21:39:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 568.141824ms core uplink us-select-1 us1-placement-20

1
...
13
14
15
16
17
...
2917