Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.............................X...] duration: 19.167380633s FAIL (2025-06-23 01:22:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.167408847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....................X.......] duration: 23.82956232s FAIL (2025-06-23 01:21:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.829594283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....................X......] duration: 52.339220869s FAIL (2025-06-23 01:21:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.339254044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X..........................X.............................X..] duration: 17.692284671s FAIL (2025-06-23 01:20:55 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.692380762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....................X.....] duration: 45.763718787s FAIL (2025-06-23 01:20:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.763748672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................X.............................X.] duration: 21.842689595s FAIL (2025-06-23 01:19:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 21.842725167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................X.............................X] duration: 1m0.026047035s FAIL (2025-06-23 01:19:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== nil> Jun 23 01:18:36.741346057 (104h55m12.304669068s) [upload=83962]: Starting upload Jun 23 01:18:36.741457026 (104h55m12.304780037s) [upload=83962]: Done starting upload Jun 23 01:18:36.741574186 (104h55m12.304897176s) [upload=83962]: Uploading segments... Jun 23 01:18:36.741945356 (104h55m12.305268357s) [upload=83962]: Got next segment. Inline: false Jun 23 01:18:36.742017220 (104h55m12.305340209s) [upload=83962]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 01:18:36.742022591 (104h55m12.305345591s) [upload=83962]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 01:18:36.777635436 (104h55m12.34095844s) [seg_pos={0 0}, upload=83962]: Begin upload segment... Jun 23 01:18:36.777700720 (104h55m12.341023723s) [seg_pos={0 0}, upload=83962]: Done begin upload segment. Jun 23 01:18:36.777707373 (104h55m12.341030382s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777724726 (104h55m12.341047737s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.777804915 (104h55m12.341127924s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778210171 (104h55m12.341533182s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778614738 (104h55m12.341937746s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.778987954 (104h55m12.342310963s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779391219 (104h55m12.342714228s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779521731 (104h55m12.342844734s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.779962650 (104h55m12.34328566s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780465195 (104h55m12.343788212s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.780615746 (104h55m12.343938757s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781165217 (104h55m12.344488225s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.781550748 (104h55m12.344873758s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782014436 (104h55m12.345337441s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782496184 (104h55m12.345819196s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.782881896 (104h55m12.346204905s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783409077 (104h55m12.346732086s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.783797923 (104h55m12.34712093s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Uploading piece... Jun 23 01:18:36.863659010 (104h55m12.426982006s) [upload=83962]: Next returned nil segment Jun 23 01:18:36.863678971 (104h55m12.427001984s) [upload=83962]: Waiting for error group managing segments... Jun 23 01:18:36.873250869 (104h55m12.436573877s) [noise=true , address=77.90.31.18:4825 , piece_id=DIQKUPGRMOVTGWHU5TVS4O2DXIBO74SKVFU3WHJN5PDU4R4XZI7Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.896075025 (104h55m12.459398017s) [noise=true , address=77.90.31.14:4695 , piece_id=RLOHBVSNWCGZU4YE2HNCBKV6V7BY7JDCDKJFWCM4YYFH5EZXWCFQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:36.930177382 (104h55m12.493500388s) [noise=true , address=104.36.82.14:6228 , piece_id=QMSNWZWKIZ4DQPXOZJC67HOTDUBWJNKY45M5QZF7I7ENC4TZE2VA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:6228): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:36.953640851 (104h55m12.51696386s) [noise=true , address=104.36.82.20:6444 , piece_id=CWBFFDJCLPZFFSP7NVOKELPXCR3T3DV43FGSSWNGJO6W2IHXEJ3A, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:6444): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.109817807 (104h55m12.673140818s) [noise=true , address=104.36.82.22:6530 , piece_id=ZHRUIZ5BZCM67DGFQZS6NI2DV732UCVDYV3GM55EISACHUWN5ZLQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:6530): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.110741515 (104h55m12.674064529s) [noise=true , address=79.127.205.234:5591 , piece_id=U2VCI7JO4IAU4B3LXOGMLQT452CIP2LLT2T53HBQFCR6U6UDQ2RA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:1ZK9ZUTAh12ywDRBtr54pdiWZi5WwkHXTmZkgoh6SpwvLAJTRi, address:79.127.205.234:5591): protocol: storage node overloaded, request limit: 2000 Jun 23 01:18:37.860754351 (104h55m13.424077339s) [noise=true , address=104.36.82.7:5933 , piece_id=Q27EIDAGLD5IZGDJ6QBB3ZA5ULFSIPXY4JWFTNBMLX346CK2ADBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.383822537 (104h55m13.947145526s) [noise=true , address=45.91.116.8:5050 , piece_id=6HAF64M3236TINYRYVIYSG27BF2QW6XGCIKE6SBVVZ527LFOP2ZQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.848556195 (104h55m14.411879183s) [noise=true , address=104.36.82.34:6569 , piece_id=EGMJEZMPGKMR5JHGP3TH3MJ7SJ5XWVMTVXCAFDX4RKPIVVMP4BAA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:38.877262383 (104h55m14.440585376s) [noise=true , address=104.36.82.51:4084 , piece_id=DODGPPX2WWL6OWRP56V6QX2JZ2PMGHXG3AEQEBRACLAHLWGJTJWA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:39.590142305 (104h55m15.15346531s) [noise=true , address=45.91.116.9:5277 , piece_id=U7K7GEX3GPXNULZP35E65E52YL2BUNEXEKTRRZUP6GAALWWPS6KA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.397250467 (104h55m16.960573472s) [noise=true , address=104.36.82.55:4225 , piece_id=M77UT6JBGUAVSXFMDUMYPRQWN4X7TOSX35LIWATYKMDXE6CSYPDA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:41.578659382 (104h55m17.141982372s) [noise=true , address=104.36.82.15:6286 , piece_id=ETSDIEVDHOYGJC62M3M6JUQQLZL5PWPYRLUAVWHU7HE7FGX4YE6Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.086228467 (104h55m19.649551478s) [noise=true , address=79.127.219.38:4026 , piece_id=EQUPNI4CUJP6MTH6MODLH6M5YJ4ZNO5ALN6GPSX22RNBCRRIPUOQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:44.523443200 (104h55m20.086766195s) [noise=true , address=77.90.31.10:4535 , piece_id=BORXDT7QJFNH6WJR6GEMRXATEOJFLESKT5XMWY6UOER6R2RW6KBQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:18:46.233887646 (104h55m21.797210646s) [noise=true , address=104.36.82.56:4245 , piece_id=CGUQ357JLAQJCIDNQ23BNV4Q3R4RGVJI2Q5Z7B56LJKZIFENJM3Q, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:21.357923809 (104h55m56.921246809s) [noise=true , address=104.36.82.54:4157 , piece_id=ESGS5AM5NBIEFUH7GGK7QV3PIQDMBLIKLIIGEBO7TJML72TF4EAQ, seg_pos={0 0}, upload=83962]: Done uploading piece. err: <nil> Jun 23 01:19:36.705243994 (104h56m12.268566993s) [noise=true , address=104.36.82.53:4128 , piece_id=YZHGLRKEISP3W4KUK3LL3EQEJUMX2LFPIDX2O676LS3GY3PNZ4TA, seg_pos={0 0}, upload=83962]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: expected piece hash; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; write tcp 159.223.143.39:39324->104.36.82.53:4128: use of closed network connection; EOF Jun 23 01:19:36.705278262 (104h56m12.26860127s) [seg_pos={0 0}, upload=83962]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 01:19:36.728720635 (104h56m12.292043639s) [upload=83962]: Done uploading segments... Jun 23 01:19:36.728814444 (104h56m12.292137449s) [upload=83962]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.026071373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.....................X....] duration: 18.752319206s FAIL (2025-06-23 01:19:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.752424604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.....................X...] duration: 30.646554683s FAIL (2025-06-23 01:18:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.646570916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X..........................X.............................] duration: 32.790372686s FAIL (2025-06-23 01:18:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.790494925s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.....................X..] duration: 44.127793227s FAIL (2025-06-23 01:18:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.127825494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X..........................X............................] duration: 20.185086833s FAIL (2025-06-23 01:17:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.185119877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.....................X.] duration: 22.573049379s FAIL (2025-06-23 01:17:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.573067746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.....................X] duration: 1m0.000642986s FAIL (2025-06-23 01:17:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00068404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X..........................X...........................] duration: 9.663579008s FAIL (2025-06-23 01:16:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.663601797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X..........................X..........................] duration: 46.365157087s FAIL (2025-06-23 01:16:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 46.365193163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 15.189212408s FAIL (2025-06-23 01:15:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.189245749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X..........................X.........................] duration: 13.479502159s FAIL (2025-06-23 01:15:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.479539384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 16.07105841s FAIL (2025-06-23 01:15:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.071096408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........X..........................X........................] duration: 20.459878926s FAIL (2025-06-23 01:14:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.459902911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................X.......................] duration: 37.479934774s FAIL (2025-06-23 01:14:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 37.479966681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 9.175574559s FAIL (2025-06-23 01:13:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.175597536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................X......................] duration: 14.813372379s FAIL (2025-06-23 01:13:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.813405828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 12.056827934s FAIL (2025-06-23 01:12:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.056857592s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................X.....................] duration: 5.124238417s FAIL (2025-06-23 01:12:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.124291212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 4.720912455s FAIL (2025-06-23 01:12:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.720936557s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................X....................] duration: 11.419506786s FAIL (2025-06-23 01:12:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.419545187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............X..........................X...................] duration: 19.61505096s FAIL (2025-06-23 01:11:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.615103266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 25.847675434s FAIL (2025-06-23 01:11:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.847690028s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............X..........................X..................] duration: 17.51139462s FAIL (2025-06-23 01:11:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.511479958s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 20.616904286s FAIL (2025-06-23 01:11:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.61693057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............X..........................X.................] duration: 12.65691195s FAIL (2025-06-23 01:10:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.656963886s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 17.019365398s FAIL (2025-06-23 01:10:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.019398695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................X..........................X................] duration: 16.434538332s FAIL (2025-06-23 01:09:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.434590262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 11.396692791s FAIL (2025-06-23 01:09:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.396725498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 15.405231237s FAIL (2025-06-23 01:09:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.405317841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................X..........................X...............] duration: 15.84688919s FAIL (2025-06-23 01:08:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.846915981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................X..........................X..............] duration: 28.157178381s FAIL (2025-06-23 01:08:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.157211349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 29.501696617s FAIL (2025-06-23 01:08:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.501729814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................X..........................X.............] duration: 13.014702166s FAIL (2025-06-23 01:07:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.014754298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................X..........................X............] duration: 30.756802393s FAIL (2025-06-23 01:07:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.756846518s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 26.874251289s FAIL (2025-06-23 01:07:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.874275445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..........................X...........] duration: 9.407195644s FAIL (2025-06-23 01:06:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.407228233s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X.................................................X.........] duration: 11.124335973s FAIL (2025-06-23 01:05:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.124389116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................X..........................X..........] duration: 5.396111132s FAIL (2025-06-23 01:05:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.39621895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................................X........] duration: 21.380944245s FAIL (2025-06-23 01:05:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.380999386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..........................X.........] duration: 20.002084417s FAIL (2025-06-23 01:05:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.002118224s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................................X.......] duration: 10.470599297s FAIL (2025-06-23 01:04:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.470664658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................................X......] duration: 17.279305988s FAIL (2025-06-23 01:03:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.279347062s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................X..........................X........] duration: 16.323950102s FAIL (2025-06-23 01:03:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 04h33m2.057393255s) [upload=83654]: Done uploading segments... Jun 23 00:56:26.494092266 (104h33m2.057415266s) [upload=83654]: Upload finished. err: <nil> Jun 23 00:57:06.637959063 (104h33m42.201282071s) [upload=83670]: Starting upload Jun 23 00:57:06.638059782 (104h33m42.201382794s) [upload=83670]: Done starting upload Jun 23 00:57:06.638215074 (104h33m42.20153807s) [upload=83670]: Uploading segments... Jun 23 00:57:06.638492617 (104h33m42.201815616s) [upload=83670]: Got next segment. Inline: false Jun 23 00:57:06.638604623 (104h33m42.201927623s) [upload=83670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 23 00:57:06.638612581 (104h33m42.201935643s) [upload=83670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 23 00:57:06.679174749 (104h33m42.242497748s) [seg_pos={0 0}, upload=83670]: Begin upload segment... Jun 23 00:57:06.679286722 (104h33m42.242609732s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679334683 (104h33m42.242657687s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.679851990 (104h33m42.243174998s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.680354309 (104h33m42.243677305s) [seg_pos={0 0}, upload=83670]: Done begin upload segment. Jun 23 00:57:06.680384815 (104h33m42.243707826s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.682638096 (104h33m42.24596111s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683110165 (104h33m42.246433164s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683325714 (104h33m42.246648726s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.683837029 (104h33m42.24716005s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.685357204 (104h33m42.248680272s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.688204648 (104h33m42.251527658s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.689270936 (104h33m42.252593945s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.690074161 (104h33m42.253397171s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.691796145 (104h33m42.25511915s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.692910883 (104h33m42.256233893s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693283259 (104h33m42.256606269s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.693680616 (104h33m42.257003626s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694005010 (104h33m42.257328021s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.694796570 (104h33m42.258119581s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Uploading piece... Jun 23 00:57:06.776383419 (104h33m42.339706418s) [upload=83670]: Next returned nil segment Jun 23 00:57:06.776399031 (104h33m42.339722033s) [upload=83670]: Waiting for error group managing segments... Jun 23 00:57:06.785181324 (104h33m42.348504319s) [noise=true , address=79.127.220.98:4429 , piece_id=BKCCICUYNS53QBKGCLPVKKFEWPYLSPS35IDPS5LUDG5LECQK63XQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1Mu1YUvXF3dbUVA5BbYEeNMXXv2dpVxGVBQazfWnmDDBZWCHoj, address:79.127.220.98:4429): protocol: storage node overloaded, request limit: 500 Jun 23 00:57:06.842506353 (104h33m42.405829353s) [noise=true , address=156.146.43.225:4448 , piece_id=4R4NXG42YZB2EMZKXRDNOUX2AV7YM4BA2OLV2T357A6N4ACLY4JA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.044437497 (104h33m42.607760488s) [noise=true , address=45.91.116.10:5084 , piece_id=JIVBBIQQZ6FFKGX4GTNTP2GZ2JBWUEKTAUHVY5LKMVHDSV64VU3Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:07.399475214 (104h33m42.962798221s) [noise=true , address=79.127.205.235:5688 , piece_id=FTVLZFWMFGPEGTVBCXJ2JWRTSBMHFGX54L6MMYTNJJXMG3J63YOQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1oUL8CDDv4TJj7nb6MYaiAGK2XwNKc6yTU4sKqsELZrKfZTwXd, address:79.127.205.235:5688): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.520844578 (104h33m43.084167585s) [noise=true , address=45.91.116.9:5288 , piece_id=Y56ILY4K55ZNJYLBYIB5LJUAZHW2337EZCOUR667HPGAIMAO5A4Q, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX, address:45.91.116.9:5288): protocol: storage node overloaded, request limit: 2000 Jun 23 00:57:07.543545968 (104h33m43.106868963s) [noise=true , address=79.127.220.97:4368 , piece_id=NT3DJMVOWCEZXJLS7Z7WBXOR43ATEVVN2QLDZAFOFI7UJ5ADVLBA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.420361791 (104h33m43.98368478s) [noise=true , address=104.36.82.12:6153 , piece_id=SXPQ3VGTTRJBG7JAP6P65U4ZRABFJ6GNVHVEO6WBLBCK7OWQD3FQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.427929362 (104h33m43.99125235s) [noise=true , address=79.127.219.38:4036 , piece_id=7L66K73MBGBYMAD6NSLFN6YARB4Y6E5Y4CUZ2GH4MA3BTRP576LA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:08.975537935 (104h33m44.538860921s) [noise=true , address=104.36.82.11:6132 , piece_id=N5A7MGWRTY4LDTZWELGLDRJI4ATWQEWJMD5GMSFO3YOFYRCP3JHQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:10.828330224 (104h33m46.391653209s) [noise=true , address=104.36.82.56:4250 , piece_id=RTHHXSTR3I54G3LZVUCAQI4ATI6C26SVR4MMPIBNK5S5VGPPXHPQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:12.862876340 (104h33m48.426199329s) [noise=true , address=104.36.82.53:4132 , piece_id=YYMHWPRVWQIRGN25DEUYGFNGBYVPF4FBWU7XIMI4IXFI4W4SPKGQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:13.438035444 (104h33m49.00135843s) [noise=true , address=104.36.82.10:6096 , piece_id=LA3EETXKVFMPSLZ5DWPRJVLL4STRYDJZ7I44LQDM6VWFESBGRKYA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:14.089960058 (104h33m49.653283049s) [noise=true , address=104.36.82.9:6021 , piece_id=47OGYLXSX2MSBXCK6MGY5JBTMU44KWOYTIGTFXMZNHQP32ZJBTWA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:18.604225370 (104h33m54.16754837s) [noise=true , address=77.90.31.14:4681 , piece_id=CFG2TF4MTVSQ3WZQTHDJ4XY4NSXWMDI3S64564ZSGVKF4LRHIJZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:19.634054264 (104h33m55.19737726s) [noise=true , address=104.36.82.54:4171 , piece_id=BO4EJJKV74AVHAMTK6G4JB3YTEZTVOQ6TRWQHH7NCPFR5J6GFDDA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:20.699198881 (104h33m56.262521882s) [noise=true , address=104.36.82.41:6781 , piece_id=BOGATUUFM4SYOKEGITTECQJZLKR7W6QIH44P3TYJK44IF3K3L3NA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:57:24.479265197 (104h34m0.042588187s) [noise=true , address=104.36.82.42:6814 , piece_id=ITUDFV2ZMN6ATAYFX2AUPWFD5VV2TXEQQ6433OFFTYOGZW3KW3ZQ, seg_pos={0 0}, upload=83670]: Done uploading piece. err: <nil> Jun 23 00:58:06.597515743 (104h34m42.160838743s) [noise=true , address=104.36.82.3:5795 , piece_id=BIPPNWMO4FZSVJEEJKWTYV6A66Y5EWXYYE4YG5ZIYUHEVTYSZKPA, seg_pos={0 0}, upload=83670]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:5795): protocol: expected piece hash; context deadline exceeded; EOF Jun 23 00:58:06.597561809 (104h34m42.160884811s) [seg_pos={0 0}, upload=83670]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jun 23 00:58:06.623438909 (104h34m42.186761901s) [upload=83670]: Done uploading segments... Jun 23 00:58:06.623496060 (104h34m42.186819071s) [upload=83670]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.323977571s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)