Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 145769 reports (Page 14 of 2916)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........X.................X..............................]
duration: 
642.368278ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:47:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 642.396655ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X.....................X........................]
duration: 
436.794339ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:47:44 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 436.822821ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.......X............X.......X...............................]
duration: 
496.506723ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:47:41 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 496.563547ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...........................X.....X..........................]
duration: 
386.659417ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:47:36 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 386.685317ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............X.................X.............................]
duration: 
405.835933ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:47:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 405.86596ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [........................X.......X...........................]
duration: 
551.4167ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:47:20 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 551.444311ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [............X..........X....................................]
duration: 
669.977262ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:47:09 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 670.085412ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [............................X.....X.........................]
duration: 
389.469009ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:46:55 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 389.496175ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [.........................X.......X..........................]
duration: 
486.756081ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:46:52 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 486.785183ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [X.............X.....................X.......................]
duration: 
459.841524ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:46:39 UTC) 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 459.873581ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X.................X............................]
duration: 
677.443158ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:46:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 677.467262ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [........X............X.......X..............................]
duration: 
499.884382ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:46:28 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 499.912519ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.............................X.....X........................]
duration: 
434.298939ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:46:19 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 434.329695ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [.............X..........X...................................]
duration: 
832.206515ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:46:17 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 832.29463ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [..........................X.......X.........................]
duration: 
475.746144ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:45:59 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 475.772165ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X..........X..................................]
duration: 
1.017201304s
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:45:51 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.017231981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.........X............X.......X.............................]
duration: 
549.345735ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:45:48 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 549.992317ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.X.............X.....................X......................]
duration: 
499.276822ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:45:46 UTC) 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 499.299711ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............X.................X...........................]
duration: 
695.040047ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:45:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 695.069445ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X..........X.................................]
duration: 
647.119363ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:45:16 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 647.143816ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [...........................X.......X........................]
duration: 
500.731751ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:45:10 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 500.760121ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..............................X.....X.......................]
duration: 
420.104479ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:45:09 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 420.195053ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............X.................X..........................]
duration: 
522.201684ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:45:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 522.230073ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..........X............X.......X............................]
duration: 
472.780623ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:45:06 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 472.808538ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................X.................X.........................]
duration: 
478.893915ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:44:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 478.923679ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..X.............X.....................X.....................]
duration: 
494.014093ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:44:35 UTC) 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 494.038218ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...........X............X.......X...........................]
duration: 
506.98406ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:44:31 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 507.072978ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.................X.................X........................]
duration: 
493.62679ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:44:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 493.65279ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [............................X.......X.......................]
duration: 
713.93948ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:44:15 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 713.980237ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............................X.....X......................]
duration: 
383.73494ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:44:14 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 383.765948ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [................X..........X................................]
duration: 
597.60093ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:44:03 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 597.626807ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [..................X.................X.......................]
duration: 
1.604236926s
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:43:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 1.604265021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [............X............X.......X..........................]
duration: 
482.654652ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:43:43 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 482.684631ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...X.............X.....................X....................]
duration: 
442.018752ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:43:39 UTC) 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 442.046607ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...................X.................X......................]
duration: 
477.554694ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:43:33 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 477.580533ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [.................X..........X...............................]
duration: 
582.854629ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:43:20 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 582.879241ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [.............................X.......X......................]
duration: 
492.614045ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:43:18 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 492.686162ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.............X............X.......X.........................]
duration: 
568.900283ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:43:06 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 568.917742ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [....................X.................X.....................]
duration: 
473.134574ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:43:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 473.157537ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [..................X..........X..............................]
duration: 
591.973383ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:43:02 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 592.015135ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [................................X.....X.....................]
duration: 
491.180543ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:42:59 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 491.208435ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....X.............X.....................X...................]
duration: 
428.434979ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:42:55 UTC) 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 428.473948ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.....................X.................X....................]
duration: 
507.905005ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:42:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 507.934092ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [..............................X.......X.....................]
duration: 
456.445752ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:42:19 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 456.525604ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....X.............X.....................X..................]
duration: 
572.347824ms
trace: 
https://jaeger.dev.storj.io/trace/2a65f5f21f735664
FAIL (2025-10-31 21:42:07 UTC) 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:27:37.767075287 (33h48m8.431024269s) [upload=35139]: Starting upload
Oct 31 21:27:37.767151190 (33h48m8.431100175s) [upload=35139]: Done starting upload
Oct 31 21:27:37.767248446 (33h48m8.431197414s) [upload=35139]: Uploading segments...
Oct 31 21:27:37.767420096 (33h48m8.431369065s) [upload=35139]: Got next segment. Inline: false
Oct 31 21:27:37.767453398 (33h48m8.43140238s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:37.767458235 (33h48m8.431407215s) [upload=35139]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:37.796680679 (33h48m8.460629658s) [seg_pos={0 0}, upload=35139]: Begin upload segment...
Oct 31 21:27:37.796741555 (33h48m8.460690522s) [seg_pos={0 0}, upload=35139]: Done begin upload segment.
Oct 31 21:27:37.796775947 (33h48m8.460724937s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.796834753 (33h48m8.460783741s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797485910 (33h48m8.461434902s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.797562041 (33h48m8.461511038s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798120963 (33h48m8.462069953s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.798626987 (33h48m8.462575978s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799167956 (33h48m8.463116938s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799454246 (33h48m8.463403227s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799567427 (33h48m8.463516407s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799793781 (33h48m8.463742775s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.799958643 (33h48m8.463907625s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800142523 (33h48m8.464091505s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800406481 (33h48m8.46435547s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800547664 (33h48m8.464496644s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.800923695 (33h48m8.464872678s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.801358082 (33h48m8.46530706s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Uploading piece...
Oct 31 21:27:37.867498510 (33h48m8.531447481s) [upload=35139]: Next returned nil segment
Oct 31 21:27:37.867511242 (33h48m8.531460213s) [upload=35139]: Waiting for error group managing segments...
Oct 31 21:27:37.874886097 (33h48m8.53883508s) [noise=true , address=145.79.227.18:4006   , piece_id=B4PUWQWDIQV7SRNRVBM7QB7Z2Y3ZFOZMVZCTPOE5DFHYPVYULBHA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.874982181 (33h48m8.538931158s) [noise=true , address=145.79.227.11:4731   , piece_id=H2DA7JHNB6ZHNW2DULHSX4MKOTMVUU6L7IF7H6SZUWY7HTQ6B6ZQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876070783 (33h48m8.540019766s) [noise=true , address=145.79.227.31:4799   , piece_id=M6XBLH4S7B6IUCTDJRUJXCH4S3HLS4TQNLCDR7Y4O4H4D7SCYZ4Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876237208 (33h48m8.540186195s) [noise=true , address=145.79.227.31:4798   , piece_id=LD64R4FXYBVF6GPOOU5WSZYHIXWNSBWJC36GE5TRNOMV3NXOPNXA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876654124 (33h48m8.540603104s) [noise=true , address=145.79.227.14:4840   , piece_id=4DG4ZVEPAQSYBUOO4RWFKI3EEHG7ULKEZMTJKTBK6LOUFT6JNXIA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.876970212 (33h48m8.540919198s) [noise=true , address=145.79.227.14:4837   , piece_id=BNPQTZUZUKDBA77HLKCXBC5E6KJL7NZPVE7H47PCHSVJE2GC7F7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877333782 (33h48m8.541282764s) [noise=true , address=145.79.227.20:4439   , piece_id=HNYWB7SNQW5RQJ73S6HXL2YHYCRLMYCGNO4WVFUBZY4HRNUL4GYQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.877880901 (33h48m8.54182988s) [noise=true , address=145.79.227.25:4618   , piece_id=G2OLQWARF4PEUYGVAWRVFUJSV7W3R7OUE7VVPEWUFILWMKYZ3U4A, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.878373488 (33h48m8.542322468s) [noise=true , address=145.79.227.23:4546   , piece_id=PDZVQ3DM3MM2VGMSB4ARQXWDEEZWYYPWXGE6EG5IAC4DP7B3JKMA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.879589817 (33h48m8.543538802s) [noise=true , address=145.79.227.15:4874   , piece_id=I4OI2I2P67LV3NEXDNRPV5JKD7STP4QWPGFI56QEO6UF6DVDLMMQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:27:37.883982183 (33h48m8.547931162s) [noise=true , address=145.79.227.5:4223    , piece_id=GBEEWTET7KCOOCVJLZLN3WG7L2JONFPAAX4JVJZDCBIO4ECJT37Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: <nil>
Oct 31 21:28:07.732361788 (33h48m38.396310777s) [noise=true , address=77.90.31.15:4698     , piece_id=6HOXHZ6ANNSQGE6DSRMVITGWWVOYZBUFCG3HE6Q5LBX24KS7UTLA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:48852->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732437508 (33h48m38.396386497s) [noise=true , address=77.90.31.18:4807     , piece_id=4ZL3AQW2GEBVVUH42L6XDAJBDU7Q5W7YHC55MVTZ6FD76HAMZSBQ, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732599272 (33h48m38.396548262s) [noise=true , address=77.90.31.17:4770     , piece_id=2DUVQHU36CUUSMLGIE7IRX4YGHZIXUBUDGG5EHISZO2EFMB2XDVA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: read tcp 159.223.143.39:43556->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732653075 (33h48m38.396602065s) [noise=true , address=45.91.116.4:4939     , piece_id=CM3QJ53SJOGNMK4MY2BKPCPNUBWI2HX7YSLUHIZQDVD62ZH7KITA, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:28:07.732776617 (33h48m38.396725605s) [noise=true , address=77.90.31.18:4806     , piece_id=DED5MA3MQNACAWOIXBDUEMAZQMYQAYYDWKNSN2HP7KTI3DQGOJ7Q, seg_pos={0 0}, upload=35139]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:52278->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.732822798 (33h48m38.396771764s) [seg_pos={0 0}, upload=35139]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:28:07.756117028 (33h48m38.420066011s) [upload=35139]: Done uploading segments...
Oct 31 21:28:07.756148948 (33h48m38.420097931s) [upload=35139]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 572.371414ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [......................X.................X...................]
duration: 
511.957527ms
trace: 
https://jaeger.dev.storj.io/trace/4f1e403e35ebdf55
FAIL (2025-10-31 21:42:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: 7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ) context deadline exceeded; piecestore: (Node ID: 1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, Piece ID: QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA) context deadline exceeded; piecestore: (Node ID: 12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL, Piece ID: 43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q) context deadline exceeded; piecestore: (Node ID: 1TkHPinJgbiX4PmRfNH7wRxtDZ9rASSvbZtLxZuxbF3UaoSqFD, Piece ID: 6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ) context deadline exceeded
======LOGS======
825937748s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162329968 (33h48m11.82627895s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.162691669 (33h48m11.826640654s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163086788 (33h48m11.82703577s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163456639 (33h48m11.827405609s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163506250 (33h48m11.827455239s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163916108 (33h48m11.827865077s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.163932850 (33h48m11.827881836s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164309014 (33h48m11.828257994s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.164677277 (33h48m11.82862626s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165073060 (33h48m11.829022049s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165499917 (33h48m11.829448902s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.165796353 (33h48m11.829745339s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166193996 (33h48m11.830142978s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166371175 (33h48m11.830320141s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.166690300 (33h48m11.830639285s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Uploading piece...
Oct 31 21:27:41.228003827 (33h48m11.891952805s) [upload=35141]: Next returned nil segment
Oct 31 21:27:41.228020129 (33h48m11.891969098s) [upload=35141]: Waiting for error group managing segments...
Oct 31 21:27:41.247795660 (33h48m11.911744639s) [noise=true , address=145.79.227.29:4944   , piece_id=CWCEJADSISASXSJMRXI6KTVWHZMYSQHMHC23KWBDFUZHGIXB3MEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.251579570 (33h48m11.915528549s) [noise=true , address=145.79.227.6:4286    , piece_id=CYTZKLIBVQDH4EL7Y6FQ63IOASIJ3SYSXLOZZXG5NTBBP6WLZSJQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.258064208 (33h48m11.922013191s) [noise=true , address=145.79.227.41:4111   , piece_id=3A67EQ3AYCIQOWKHOJJOHU7OWKOYE56T5APXYRAZMQHWGM7OUIEA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.264573964 (33h48m11.928522943s) [noise=true , address=145.79.227.14:4850   , piece_id=SQQWZRJL7U7VL7P3QQBIBL4MR6PVWHF4JGFDUGYRIJ454CSKLPGQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:27:41.379551499 (33h48m12.043500478s) [noise=true , address=145.79.227.13:4823   , piece_id=7I5HSCCPMNCEZB5SNZFJQACCHMADGZ4T6JNDPRLYVTJOZMAJFVFQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:06.667196919 (33h48m37.331145902s) [noise=true , address=45.91.116.7:4043     , piece_id=6JSXYATWZOOLWW2INRWNZWC7SPDH3ADZR2SNVCWJX6OKAHRIYURQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:07.361827475 (33h48m38.025776457s) [noise=true , address=77.90.31.14:4683     , piece_id=HV3QKGXKD7OKYQPNDJ5I6IUAIG7HTZTG5M3FWCR3OUYO7BSGURXA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.192730706 (33h48m39.856679686s) [noise=true , address=45.91.116.4:4956     , piece_id=CMT3YM45ZFQF4BKDAIRV4ZFS4NSUUN6CJAMCIRPLPA56MOGZWKZQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:09.820313132 (33h48m40.484262113s) [noise=true , address=77.90.31.11:4579     , piece_id=7LTIFFSFZ723K22YVZ6DALJW27JTJCFCH3OKP7J2E6D2EFGZXZOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:10.352915355 (33h48m41.01686434s) [noise=true , address=77.90.31.18:4829     , piece_id=UTA7K7KFZU3WXXPWEA6RZXLU3H6YEL43ITP6RGNBJAKUZD3FGUGA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:12.851504814 (33h48m43.515453793s) [noise=true , address=45.91.116.6:4996     , piece_id=43FK4A3XK7H3RTYRXRGPEBBV7UXGUFWZCQVIWZ2QM7EW6HDLQJ6Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.219580195 (33h49m2.883529178s) [noise=true , address=77.90.31.19:4846     , piece_id=UNWSQXCL5DYGXYTNKEJLLDXJVVDH2HMTNVIV6ZC2F6SKNQ5SEEEQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:32.305035603 (33h49m2.968984583s) [noise=true , address=77.90.31.18:4832     , piece_id=AHPBKHPUCBIPLUGEYKLNXFSYUN6M4QWFXTHWNUCHC5QRORIGMJBA, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070426064 (33h49m7.734375043s) [noise=true , address=45.91.116.10:4091    , piece_id=QMZRK4HIDQQ2SITTKVHG6EXWJOP4O5UWKRLZA4X7DEXZJBITLJKQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: <nil>
Oct 31 21:28:37.070459295 (33h49m7.734408267s) [seg_pos={0 0}, upload=35141]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:37.070675613 (33h49m7.734624592s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070683034 (33h49m7.734632017s) [noise=true , address=77.90.31.17:4785     , piece_id=VZLHISJ3AJERNPVPEXCMCIYW5UWPKYE7DU3QEDN4BHOD7F5YOZ5Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070696091 (33h49m7.734645081s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122RnVXziDxm69JuYuwoeufcUUTdXtbHART8UpbFTPbxC1w543k): protocol: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50524->77.90.31.14:4675: use of closed network connection
Oct 31 21:28:37.070710064 (33h49m7.734659057s) [noise=true , address=77.90.31.14:4675     , piece_id=ZF72DCN6RABRN5R3BE6KOGUMTTHEGL4PFGTZLHICGOQTGJMEIYOQ, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.070838767 (33h49m7.734787747s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:37.070858533 (33h49m7.734807513s) [noise=true , address=77.90.31.10:4540     , piece_id=KJOPOYYGR4NAXDJY3RE5TI2IK6WYOSP36JDMU4C6FA7AWAMI3K5A, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071223711 (33h49m7.735172696s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ): protocol: read tcp 159.223.143.39:50240->77.90.31.13:4625: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:37.071232303 (33h49m7.735181284s) [noise=true , address=77.90.31.13:4625     , piece_id=GMSPFEMEOFEYDOOYVUFZFIWAE2U3RE7JGWM2HI4YJU6WFDXKHY3Q, seg_pos={0 0}, upload=35141]: Long tail cancellation due to optimal threshold
Oct 31 21:28:37.071252378 (33h49m7.73520135s) [seg_pos={0 0}, upload=35141]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:37.071334434 (33h49m7.735283406s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:37.071350346 (33h49m7.735299326s) [upload=35141]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:37.128760544 (33h49m7.792709526s) [upload=35141]: Done uploading segments...
Oct 31 21:28:37.128778987 (33h49m7.792727966s) [upload=35141]: Upload finished. err: <nil>
Duration: 511.985221ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip: 
2 out of 60 is failed (threshold=3) [...................X..........X.............................]
duration: 
565.697028ms
trace: 
https://jaeger.dev.storj.io/trace/464ff9dd561aefd2
FAIL (2025-10-31 21:42:05 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 566.2802ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.................................X.....X....................]
duration: 
438.730932ms
trace: 
https://jaeger.dev.storj.io/trace/22de3cd654292a7d
FAIL (2025-10-31 21:42:03 UTC) 
orbiter warning: 
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 8)
======LOGS======
Oct 31 21:27:07.866208621 (33h47m38.530157602s) [upload=35132]: Starting upload
Oct 31 21:27:07.866279020 (33h47m38.530228004s) [upload=35132]: Done starting upload
Oct 31 21:27:07.866436946 (33h47m38.530385917s) [upload=35132]: Uploading segments...
Oct 31 21:27:07.866626574 (33h47m38.530575554s) [upload=35132]: Got next segment. Inline: false
Oct 31 21:27:07.866696457 (33h47m38.530645426s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:07.866701550 (33h47m38.530650521s) [upload=35132]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:07.895492270 (33h47m38.559441253s) [seg_pos={0 0}, upload=35132]: Begin upload segment...
Oct 31 21:27:07.895547528 (33h47m38.559496513s) [seg_pos={0 0}, upload=35132]: Done begin upload segment.
Oct 31 21:27:07.895575464 (33h47m38.559524446s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.895755505 (33h47m38.559704485s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.896288765 (33h47m38.560237755s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897321991 (33h47m38.561270982s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.897591022 (33h47m38.561540013s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.898706945 (33h47m38.562656129s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.899440008 (33h47m38.563388996s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900075099 (33h47m38.564024089s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900636550 (33h47m38.564585536s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.900999790 (33h47m38.564948779s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.901522607 (33h47m38.565471594s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902042338 (33h47m38.565991328s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902507679 (33h47m38.566456665s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.902961202 (33h47m38.566910192s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.903612903 (33h47m38.567561893s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904089002 (33h47m38.568037991s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.904643332 (33h47m38.568592319s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.906103999 (33h47m38.570052989s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Uploading piece...
Oct 31 21:27:07.992220811 (33h47m38.65616978s) [upload=35132]: Next returned nil segment
Oct 31 21:27:07.992234842 (33h47m38.656183812s) [upload=35132]: Waiting for error group managing segments...
Oct 31 21:27:07.998921929 (33h47m38.662870913s) [noise=true , address=145.79.227.33:4894   , piece_id=P6TXAXQCJXY6SVKMCRTB4Z5YCLJX4FY4RE4MF2ALC2MLNBCXIXNA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.000057355 (33h47m38.664006349s) [noise=true , address=145.79.227.21:4504   , piece_id=4EBOYJDY5YC4PJ65HPJWGNLWHXJQWIZTI3PEOLR4R3PC3F6XSN3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.018621423 (33h47m38.682570402s) [noise=true , address=145.79.227.5:4225    , piece_id=CW53PVYVC5H3RH4PD6NDD7WL6RFTL6XIOFIWEBUTHNRPF6OZAASA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.023827749 (33h47m38.687776736s) [noise=true , address=145.79.227.17:4990   , piece_id=5PX4HB2M3P2DAMWEBVJQY5GSRVD4VLQFUJKVMAGVKEAKSNNMZISA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.024791164 (33h47m38.688740145s) [noise=true , address=145.79.227.4:4188    , piece_id=ZVAEYYWYNQJ4AMTWBBYEFYXOZHA5336MZWY6IPSM6DD6KQBRN7SA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.028084513 (33h47m38.692033498s) [noise=true , address=145.79.227.14:4859   , piece_id=57MGAGFTM63JFC3V2TZ5AUR2HG7DUAHE3SJTQ3H5RELFCU7LARSA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:27:08.682982151 (33h47m39.346931142s) [noise=true , address=145.79.227.6:4267    , piece_id=B5O3CWYSC4MEHGALX4LNISAINLRJ5YVAUHJUW4O34RVKS3Q3RK4A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:06.810422560 (33h48m37.474371544s) [noise=true , address=77.90.31.11:4576     , piece_id=LS3JZV6RUZWWSQ7UNJ3UVDBNWRGIRQ5QPFK35PZOX6GOWYSWTZ3Q, seg_pos={0 0}, upload=35132]: Done uploading piece. err: <nil>
Oct 31 21:28:07.832600724 (33h48m38.496549706s) [noise=true , address=45.91.116.9:4277     , piece_id=DKYBEXRP4AAGGHPZ34F2VT7LEBW7VILU7JDYPBC6QH3KLRDNEARQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: write tcp 159.223.143.39:34186->45.91.116.9:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832797728 (33h48m38.496746709s) [noise=true , address=45.91.116.6:4000     , piece_id=2YR4K6ZPQ6A2VYYIQXSQRUITQXQ4FJ3STRBOAIUQMISG4QZCKTMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: read tcp 159.223.143.39:41618->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832851666 (33h48m38.496800653s) [noise=true , address=45.91.116.7:4016     , piece_id=4AX2NEOYSMTX5NJ5MMSGSVPJMNJ4UIINGZ2GHQNISQOEHKGH5MZA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1GeydXGpXPZzL65aqsaGiRQjz7NRzL9BZCUb5MEUWCAzNPMRPW, address:45.91.116.7:4016): protocol: rpc: dial tcp 45.91.116.7:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.832963615 (33h48m38.496912593s) [noise=true , address=77.90.31.19:4856     , piece_id=MPH5QDPMDGDG6PYAETBJME6OTDXHDL3JHURDAYFEG3LOHOWQFB7A, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: read tcp 159.223.143.39:43904->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833341948 (33h48m38.497298048s) [noise=true , address=45.91.116.10:4093    , piece_id=7ZWBP3LEDUPF4Q75TFSCBQCF4A52RB4OUXSZTOCWJOXZRAC3Z6JQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: read tcp 159.223.143.39:54654->45.91.116.10:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833521809 (33h48m38.497470776s) [noise=true , address=77.90.31.13:4652     , piece_id=RBIMJEOP56S24ZLP2WYG3IGFEOIL77D65EHGYL73HS57IGY5TQSQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA, address:77.90.31.13:4652): protocol: read tcp 159.223.143.39:43334->77.90.31.13:4652: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833561449 (33h48m38.497510444s) [noise=true , address=77.90.31.12:4605     , piece_id=GQH2V2JYEIJIAI2ZVDM77K236R5CDFTPLNLJA7W7WZCHYJFBA7RQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: read tcp 159.223.143.39:49408->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833743421 (33h48m38.497692407s) [noise=true , address=45.91.116.8:4078     , piece_id=ODABCHGVB4IJKXNXXFJH4JRFDJ73H3HXE33AWLICCTASGFNXU6UQ, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:12cz8jq43fTVXSkjyjTxHmpzbTSvCcNQJxJ177qFzqZNq9b2dE1, address:45.91.116.8:4078): protocol: read tcp 159.223.143.39:45276->45.91.116.8:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.833927320 (33h48m38.497876302s) [noise=true , address=45.91.116.11:4151    , piece_id=3RURLMX35FAPXR6WE6OFJLHLWB2KMBOKYYMQFUOSVBU76EF2ZNMA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:13V8hQxkV4kuinaNsJzW8FYqUaRWrsrLy4FG9UtanaT7F6jkgC, address:45.91.116.11:4151): protocol: read tcp 159.223.143.39:48596->45.91.116.11:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834067425 (33h48m38.49801641s) [noise=true , address=45.91.116.12:4235    , piece_id=L4OLHEZ73HH46FQQ2JLW3JKZO5JZD2TRWDR5YWLUTLMOB5QQVLCA, seg_pos={0 0}, upload=35132]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:38334->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:07.834091027 (33h48m38.498040006s) [seg_pos={0 0}, upload=35132]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil>
Oct 31 21:28:07.858685769 (33h48m38.522634749s) [upload=35132]: Done uploading segments...
Oct 31 21:28:07.858719000 (33h48m38.52266798s) [upload=35132]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8)
Duration: 438.759886ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [X.............X............X.......X........................]
duration: 
452.688907ms
trace: 
https://jaeger.dev.storj.io/trace/7fc66a580bd3dc91
FAIL (2025-10-31 21:42:00 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.736626ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
2 out of 60 is failed (threshold=6) [...............................X.......X....................]
duration: 
478.836948ms
trace: 
https://jaeger.dev.storj.io/trace/689f32b5a76d000d
FAIL (2025-10-31 21:41:49 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 478.902674ms core uplink us-select-1 us1-placement-20