Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 165387 reports (Page 3083 of 3308)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................................X..XX.XXXXX........] duration: 445.807878ms FAIL (2025-07-11 22:12:50 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 445.837849ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................................XX..XXXX......] duration: 615.345067ms FAIL (2025-07-11 22:12:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 615.387241ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 702.245521ms FAIL (2025-07-11 22:12:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 702.278893ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................................X..XX.XXXXX.......] duration: 472.197659ms FAIL (2025-07-11 22:12:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 472.230776ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 697.034157ms FAIL (2025-07-11 22:12:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 697.085995ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................................XX..XXXX.....] duration: 680.126526ms FAIL (2025-07-11 22:11:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 680.153275ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................................X..XX.XXXXX......] duration: 621.50165ms FAIL (2025-07-11 22:11:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 621.535959ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 753.625588ms FAIL (2025-07-11 22:11:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 753.711613ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............................................X..XX.XXXXX.....] duration: 744.949243ms FAIL (2025-07-11 22:11:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 744.991852ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.465250185s FAIL (2025-07-11 22:11:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.465301229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 1.260167745s FAIL (2025-07-11 22:11:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1.260230196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................................XX..XXXX....] duration: 683.457967ms FAIL (2025-07-11 22:11:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 683.517483ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 936.251857ms FAIL (2025-07-11 22:10:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 936.304214ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............................................X..XX.XXXXX....] duration: 538.212051ms FAIL (2025-07-11 22:10:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 538.244302ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.380174633s FAIL (2025-07-11 22:10:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.380198354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............................................X..XX.XXXXX...] duration: 1.499804193s FAIL (2025-07-11 22:09:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.499885693s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 1.471737295s FAIL (2025-07-11 22:09:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1.47176488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................................XX..XXXX...] duration: 2.86797163s FAIL (2025-07-11 22:09:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.868793231s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.249966434s FAIL (2025-07-11 22:09:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.250014812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............................................X..XX.XXXXX..] duration: 8.595960535s FAIL (2025-07-11 22:09:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.596008032s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX..XXXX..] duration: 3.741129485s FAIL (2025-07-11 22:09:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.741206585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................................................X..XX.XXXXX.] duration: 25.424540686s FAIL (2025-07-11 22:09:21 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 25.424572964s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................XX..XXXX.] duration: 12.185156328s FAIL (2025-07-11 22:09:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.185187512s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...................................................XX..XXXX] duration: 30.00071558s FAIL (2025-07-11 22:08:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000741802s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 630.553888ms FAIL (2025-07-11 22:08:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 630.591954ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 619.735069ms FAIL (2025-07-11 22:08:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 619.786613ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................................................X..XX.XXXXX] duration: 30.031304231s FAIL (2025-07-11 22:08:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.031330678s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 1.910450536s FAIL (2025-07-11 22:07:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1.910496276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.028506438s FAIL (2025-07-11 22:07:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.028522883s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX...................................................XX..XXX] duration: 30.000745453s FAIL (2025-07-11 22:07:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000788407s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.704879734s FAIL (2025-07-11 22:07:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.704910187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..................................................X..XX.XXXX] duration: 30.02907087s FAIL (2025-07-11 22:07:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ) [upload=356247]: Got next segment. Inline: false Jul 11 22:06:53.213038576 (444h55m20.139333265s) [upload=356247]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:06:53.213046522 (444h55m20.139341209s) [upload=356247]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:53.241916770 (444h55m20.16821146s) [seg_pos={0 0}, upload=356247]: Begin upload segment... Jul 11 22:06:53.241981852 (444h55m20.168276545s) [seg_pos={0 0}, upload=356247]: Done begin upload segment. Jul 11 22:06:53.242014670 (444h55m20.168309368s) [noise=true , address=104.36.82.11:4114 , piece_id=HPWVKKK3RXNJNUBGYGBB2OBL76YK7G2IABBQTX4U75TUBG7OPCWQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.242063172 (444h55m20.168357867s) [noise=true , address=104.36.82.13:4186 , piece_id=W2IEKMHUS7E7L2HGF45SD422T4XBGGHTAUTMGTOBTN4FDQ5EVMIQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.242500440 (444h55m20.168795137s) [noise=true , address=104.36.82.14:4223 , piece_id=HBBHH4GTE4P3JY5KR5L42OPSBZV7Q7G7M4L564D2TDQPEV7MLLWQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.242542265 (444h55m20.168836955s) [noise=true , address=104.36.82.5:4837 , piece_id=E2ZRVYPEPOYUIDDXFYUBAQCKN2BSAKYR5NQXJUQHXUPF3GGUVQ2A, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.242938767 (444h55m20.169233457s) [noise=true , address=104.36.82.9:4007 , piece_id=43MZ3TZPPUQZQ2HBMZG3YHLHG5JCOSP3PA6PN6JKFMCRIB63VNUQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.243082536 (444h55m20.16937724s) [noise=true , address=77.90.31.14:4661 , piece_id=7MPTD63T2EEFUTLHRZVBOXCFOLLXO7O77ZXUQ4Z5IL4KGIWGFIMQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.243334075 (444h55m20.169628761s) [noise=true , address=77.90.31.11:4555 , piece_id=TDLSD63LNZR6GBWHPWRHOCOGVHALEXYQFNWBZPCPRE44TRRJRBEQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.243530070 (444h55m20.169824771s) [noise=true , address=104.36.82.4:4803 , piece_id=7MSIWKFUIKVUANRYUKBRFNLVS6L7ZVFYMT6WB4TWBRPZEMOW34SQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.243712390 (444h55m20.170007082s) [noise=true , address=104.36.82.6:4876 , piece_id=SGCZCUVJI45J2IKLODRMMJC24MC65RHF6LGZVHTQYSZEFQ62LKGA, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.243944298 (444h55m20.170239006s) [noise=true , address=104.36.82.12:4149 , piece_id=55734NNDMDA3CS5DPF7AVCIYXMS5RBA6HHJT5OCG5Q6LEBVPUZRA, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.244181842 (444h55m20.170476539s) [noise=true , address=45.91.116.4:4937 , piece_id=BBBCGBJQVCWDGPLXZQ3RYXNQ4KEAR2NJFT6WP7LOTTQUKVEVJTRA, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.244363860 (444h55m20.170658557s) [noise=true , address=45.91.116.14:4191 , piece_id=ZZ5SNEMHIWHWD6FZDELNNBA2VQSHBF3LH2RLF46G674QNYW5VXBA, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.244743761 (444h55m20.171038456s) [noise=true , address=104.36.82.15:4259 , piece_id=D26DTTLWDTLRFFLVA3JU7D5NDNBJQLPT4FEDCQNSLHRYO47GZNBQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.245641183 (444h55m20.171935872s) [noise=true , address=77.90.31.10:4517 , piece_id=SXI4YWXUZ5MTQZRGFDV7WZSM7K2LSII2F7ZRMARIJQ4OA4TVZMCQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.247160395 (444h55m20.173455083s) [noise=true , address=45.91.116.7:4011 , piece_id=5PMCST226HOPRU2LH4A47B3HJHQTPJUJ5ZCQ2LF5W7SZY5IJTHBQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.247540219 (444h55m20.173834907s) [noise=true , address=45.91.116.5:4299 , piece_id=26X7UKTBNTNZPZPUSF47ZFH2PZHDP7HEF4BLFZXGF3WXODFEZ5HQ, seg_pos={0 0}, upload=356247]: Uploading piece... Jul 11 22:06:53.327069513 (444h55m20.253364189s) [upload=356247]: Next returned nil segment Jul 11 22:06:53.327089283 (444h55m20.253383977s) [upload=356247]: Waiting for error group managing segments... Jul 11 22:06:53.332335186 (444h55m20.258629866s) [noise=true , address=45.91.116.7:4011 , piece_id=5PMCST226HOPRU2LH4A47B3HJHQTPJUJ5ZCQ2LF5W7SZY5IJTHBQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:06:53.332756447 (444h55m20.259051127s) [noise=true , address=45.91.116.4:4937 , piece_id=BBBCGBJQVCWDGPLXZQ3RYXNQ4KEAR2NJFT6WP7LOTTQUKVEVJTRA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:06:53.339109919 (444h55m20.265404612s) [noise=true , address=45.91.116.5:4299 , piece_id=26X7UKTBNTNZPZPUSF47ZFH2PZHDP7HEF4BLFZXGF3WXODFEZ5HQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:06:53.343064572 (444h55m20.269359266s) [noise=true , address=45.91.116.14:4191 , piece_id=ZZ5SNEMHIWHWD6FZDELNNBA2VQSHBF3LH2RLF46G674QNYW5VXBA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:06:56.255721821 (444h55m23.182016508s) [noise=true , address=77.90.31.14:4661 , piece_id=7MPTD63T2EEFUTLHRZVBOXCFOLLXO7O77ZXUQ4Z5IL4KGIWGFIMQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:06:58.442542035 (444h55m25.368836721s) [noise=true , address=77.90.31.11:4555 , piece_id=TDLSD63LNZR6GBWHPWRHOCOGVHALEXYQFNWBZPCPRE44TRRJRBEQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:06:58.971425034 (444h55m25.897719706s) [noise=true , address=77.90.31.10:4517 , piece_id=SXI4YWXUZ5MTQZRGFDV7WZSM7K2LSII2F7ZRMARIJQ4OA4TVZMCQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: <nil> Jul 11 22:07:23.184435141 (444h55m50.110729838s) [noise=true , address=104.36.82.5:4837 , piece_id=E2ZRVYPEPOYUIDDXFYUBAQCKN2BSAKYR5NQXJUQHXUPF3GGUVQ2A, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.184627276 (444h55m50.110921967s) [noise=true , address=104.36.82.9:4007 , piece_id=43MZ3TZPPUQZQ2HBMZG3YHLHG5JCOSP3PA6PN6JKFMCRIB63VNUQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.184695626 (444h55m50.110990316s) [noise=true , address=104.36.82.6:4876 , piece_id=SGCZCUVJI45J2IKLODRMMJC24MC65RHF6LGZVHTQYSZEFQ62LKGA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.184765159 (444h55m50.111059847s) [noise=true , address=104.36.82.12:4149 , piece_id=55734NNDMDA3CS5DPF7AVCIYXMS5RBA6HHJT5OCG5Q6LEBVPUZRA, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.184840994 (444h55m50.111135683s) [noise=true , address=104.36.82.13:4186 , piece_id=W2IEKMHUS7E7L2HGF45SD422T4XBGGHTAUTMGTOBTN4FDQ5EVMIQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:37142->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.185226908 (444h55m50.111521598s) [noise=true , address=104.36.82.11:4114 , piece_id=HPWVKKK3RXNJNUBGYGBB2OBL76YK7G2IABBQTX4U75TUBG7OPCWQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.185304033 (444h55m50.111598707s) [noise=true , address=104.36.82.4:4803 , piece_id=7MSIWKFUIKVUANRYUKBRFNLVS6L7ZVFYMT6WB4TWBRPZEMOW34SQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: context deadline exceeded; context deadline exceeded Jul 11 22:07:23.185464926 (444h55m50.111759617s) [noise=true , address=104.36.82.14:4223 , piece_id=HBBHH4GTE4P3JY5KR5L42OPSBZV7Q7G7M4L564D2TDQPEV7MLLWQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:45784->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.185569967 (444h55m50.111864643s) [noise=true , address=104.36.82.15:4259 , piece_id=D26DTTLWDTLRFFLVA3JU7D5NDNBJQLPT4FEDCQNSLHRYO47GZNBQ, seg_pos={0 0}, upload=356247]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:35752->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:23.185590519 (444h55m50.111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.029097478s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX...................................................XX..XX] duration: 30.001965729s FAIL (2025-07-11 22:06:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00199127s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................................X..XX.XXX] duration: 30.030308492s FAIL (2025-07-11 22:06:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== ot 8) Jul 11 22:06:18.051185338 (444h54m44.977480033s) [upload=356238]: Starting upload Jul 11 22:06:18.051318121 (444h54m44.977612817s) [upload=356238]: Done starting upload Jul 11 22:06:18.051487943 (444h54m44.977782632s) [upload=356238]: Uploading segments... Jul 11 22:06:18.051923180 (444h54m44.978217876s) [upload=356238]: Got next segment. Inline: false Jul 11 22:06:18.051992272 (444h54m44.978286964s) [upload=356238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:06:18.052020128 (444h54m44.978314816s) [upload=356238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:18.105700657 (444h54m45.031995345s) [seg_pos={0 0}, upload=356238]: Begin upload segment... Jul 11 22:06:18.105763227 (444h54m45.032057916s) [seg_pos={0 0}, upload=356238]: Done begin upload segment. Jul 11 22:06:18.105789758 (444h54m45.032084447s) [noise=true , address=77.90.31.10:4518 , piece_id=DISXHPA7KYC2KKYMHKFDOL3GNNRAETGOUIBUZ3GM3TCDEHTGZOEQ, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.106022684 (444h54m45.032317362s) [noise=true , address=77.90.31.13:4627 , piece_id=5OKHNAVPMUVLBRFBN4T6WV357I2EE4F4ID4HTK5FXS6LDONEIPVA, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.106427098 (444h54m45.032721776s) [noise=true , address=104.36.82.10:4078 , piece_id=ZWTGOOBE4Y2IG3I43SD4QIXP5IU2N7SLLTSV37GAZWRSKNKBP65A, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.107423986 (444h54m45.033718677s) [noise=true , address=45.91.116.12:4227 , piece_id=V5MUOE53LDI6ZI77XSPLATYUV5BPCG7GWBLO6L2ILNRACNXPDJRA, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.108113717 (444h54m45.034408398s) [noise=true , address=104.36.82.15:4259 , piece_id=JUS4XDWTU36Z4QD57NU5XEBM5WIFBKOYFOAEBE655TJL77WJ464Q, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.108801938 (444h54m45.03509662s) [noise=true , address=104.36.82.14:4222 , piece_id=VUWXRL74AE2SQHXH6PB2JARXMCQLXKET3NNGI5PRS24ZVYR4P4DA, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.109474825 (444h54m45.035769513s) [noise=true , address=77.90.31.15:4697 , piece_id=WEHCZSJVYPCWAYUIBAFUZKXUXDGVWBKQ6K57GSJH354FOAWRQE2Q, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.110085284 (444h54m45.03637997s) [noise=true , address=104.36.82.6:4873 , piece_id=JVCBPXQR73Q3IN2VLNSTR2NR6FNTPNPSYKS7ASNDK5YR4QTWKGHA, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.110666176 (444h54m45.036960858s) [noise=true , address=45.91.116.5:4298 , piece_id=AV4HIPXRHWFDRHKRFSOCGPHKGVDJJJGPTASFYJWHJKEKLNZH4HYA, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.111303913 (444h54m45.037598594s) [noise=true , address=77.90.31.16:4735 , piece_id=JGG6BP6YJHZZUYEMOLXN5UU3NXXSRI4JQZDOVIGSU47FATVCEQNQ, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.111863776 (444h54m45.038158467s) [noise=true , address=77.90.31.17:4771 , piece_id=ZXMQJXXTXVKY3LOUVMXAQDXXSOUBKN2A53FJF6DERHWIVD7EJS2A, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.112422111 (444h54m45.038716789s) [noise=true , address=45.91.116.14:4189 , piece_id=LONAOHD7PA7GKNDEIW4GJPTZSVYRYW2T4TM3H27JBZSN7RKDJHQA, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.113012204 (444h54m45.039306885s) [noise=true , address=45.91.116.9:4261 , piece_id=S5DRNWQIIHA6Q7ASAJQ2AHV6CDEIWUIR3BW7LR5E6VZOGF55NK6Q, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.115891525 (444h54m45.042186201s) [noise=true , address=104.36.82.12:4149 , piece_id=IRTZMAJECCGE6BDPSRYRWDQIERNO7RVKB2J6Z3STLN55A453HKXQ, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.116450165 (444h54m45.042744837s) [noise=true , address=104.36.82.9:4007 , piece_id=R4XKPFVEZIPSIBS4YEGI7IZY3A5F3JT4SKKOVNFVRNXDGLI3HLOQ, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.116908391 (444h54m45.043203063s) [noise=true , address=104.36.82.14:4223 , piece_id=4FQRRH7Q5PU2UDW7G2IRZOR3MVZRDARCXXDOY7KY3DSQ7QXACMLQ, seg_pos={0 0}, upload=356238]: Uploading piece... Jul 11 22:06:18.188681218 (444h54m45.114975895s) [upload=356238]: Next returned nil segment Jul 11 22:06:18.188697258 (444h54m45.114991937s) [upload=356238]: Waiting for error group managing segments... Jul 11 22:06:18.198308565 (444h54m45.124603255s) [noise=true , address=45.91.116.12:4227 , piece_id=V5MUOE53LDI6ZI77XSPLATYUV5BPCG7GWBLO6L2ILNRACNXPDJRA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:18.243789324 (444h54m45.170084015s) [noise=true , address=45.91.116.9:4261 , piece_id=S5DRNWQIIHA6Q7ASAJQ2AHV6CDEIWUIR3BW7LR5E6VZOGF55NK6Q, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:18.257171408 (444h54m45.183466088s) [noise=true , address=45.91.116.14:4189 , piece_id=LONAOHD7PA7GKNDEIW4GJPTZSVYRYW2T4TM3H27JBZSN7RKDJHQA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:18.296692639 (444h54m45.222987326s) [noise=true , address=45.91.116.5:4298 , piece_id=AV4HIPXRHWFDRHKRFSOCGPHKGVDJJJGPTASFYJWHJKEKLNZH4HYA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:18.432435928 (444h54m45.358730608s) [noise=true , address=77.90.31.10:4518 , piece_id=DISXHPA7KYC2KKYMHKFDOL3GNNRAETGOUIBUZ3GM3TCDEHTGZOEQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:18.485254644 (444h54m45.411549343s) [noise=true , address=77.90.31.13:4627 , piece_id=5OKHNAVPMUVLBRFBN4T6WV357I2EE4F4ID4HTK5FXS6LDONEIPVA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:18.909944371 (444h54m45.836239044s) [noise=true , address=77.90.31.16:4735 , piece_id=JGG6BP6YJHZZUYEMOLXN5UU3NXXSRI4JQZDOVIGSU47FATVCEQNQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:19.558796563 (444h54m46.485091238s) [noise=true , address=77.90.31.15:4697 , piece_id=WEHCZSJVYPCWAYUIBAFUZKXUXDGVWBKQ6K57GSJH354FOAWRQE2Q, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:19.945915442 (444h54m46.872210115s) [noise=true , address=77.90.31.17:4771 , piece_id=ZXMQJXXTXVKY3LOUVMXAQDXXSOUBKN2A53FJF6DERHWIVD7EJS2A, seg_pos={0 0}, upload=356238]: Done uploading piece. err: <nil> Jul 11 22:06:48.022035645 (444h55m14.948330332s) [noise=true , address=104.36.82.6:4873 , piece_id=JVCBPXQR73Q3IN2VLNSTR2NR6FNTPNPSYKS7ASNDK5YR4QTWKGHA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022120894 (444h55m14.948415588s) [noise=true , address=104.36.82.12:4149 , piece_id=IRTZMAJECCGE6BDPSRYRWDQIERNO7RVKB2J6Z3STLN55A453HKXQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022194382 (444h55m14.948489069s) [noise=true , address=104.36.82.9:4007 , piece_id=R4XKPFVEZIPSIBS4YEGI7IZY3A5F3JT4SKKOVNFVRNXDGLI3HLOQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:47818->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022306683 (444h55m14.948601369s) [noise=true , address=104.36.82.15:4259 , piece_id=JUS4XDWTU36Z4QD57NU5XEBM5WIFBKOYFOAEBE655TJL77WJ464Q, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:52300->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022314931 (444h55m14.948609621s) [noise=true , address=104.36.82.14:4223 , piece_id=4FQRRH7Q5PU2UDW7G2IRZOR3MVZRDARCXXDOY7KY3DSQ7QXACMLQ, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:41088->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022375083 (444h55m14.94866977s) [noise=true , address=104.36.82.10:4078 , piece_id=ZWTGOOBE4Y2IG3I43SD4QIXP5IU2N7SLLTSV37GAZWRSKNKBP65A, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:39860->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022478018 (444h55m14.948772698s) [noise=true , address=104.36.82.14:4222 , piece_id=VUWXRL74AE2SQHXH6PB2JARXMCQLXKET3NNGI5PRS24ZVYR4P4DA, seg_pos={0 0}, upload=356238]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:35382->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:48.022496568 (444h55m14.948791256s) [seg_pos={0 0}, upload=356238]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 11 22:06:48.050077138 (444h55m14.976371826s) [upload=356238]: Done uploading segments... Jul 11 22:06:48.050101701 (444h55m14.976396386s) [upload=356238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030329267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.029184034s FAIL (2025-07-11 22:06:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== Begin upload segment... Jul 11 22:05:48.115011540 (444h54m15.041306216s) [seg_pos={0 0}, upload=356232]: Done begin upload segment. Jul 11 22:05:48.115076977 (444h54m15.041371666s) [noise=true , address=104.36.82.18:4369 , piece_id=NUCSK7GBSDCHWQ5GDGMGSPD677FGLPAG2NU7ZKQVBOR6ANAQCKFA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.115438577 (444h54m15.041733269s) [noise=true , address=79.127.219.38:4034 , piece_id=ALLHZPUBKX6UJKPRB2WMBTAYVWONJEDU4YJHNXT65HRBWANF4AKA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.115816062 (444h54m15.042110761s) [noise=true , address=45.91.116.14:4211 , piece_id=W5CAC3SOO3POEU2BVSVBGFYEP3XZQJUXKM5DQWE2KXTP3MVEU6ZA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.116382152 (444h54m15.042676841s) [noise=true , address=64.49.13.154:4925 , piece_id=GDEY55ZY2HESA33OHQYLQRY2IK2NS5LOPQTCXS52KYW5QGBWYGLA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.116665236 (444h54m15.042959929s) [noise=true , address=77.90.31.15:4722 , piece_id=4NSFGP3JAPTC6PK5TQT6H3N6OWPKFJTSPTEXWAULIRQLZTBQJ5CQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.116899835 (444h54m15.043194521s) [noise=true , address=104.36.82.49:4011 , piece_id=6LB5ZILMX3B6CGTJFWEWGJ6GQY4LWVL6QEB5JUHP5Z324PFGPRQQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.117372000 (444h54m15.043666686s) [noise=true , address=77.90.31.16:4757 , piece_id=2CZYVVBJQDJUQOHSGM5J6EIZKTX53GC72MF5VKZRZO6Z5C57WIQQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.117821745 (444h54m15.044116432s) [noise=true , address=77.90.31.19:4876 , piece_id=NZJZB2CJ2GJ6QYUFJC4ER3T6S22BVCRJRTQUWVYCCE7DIJKUHFWQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.122181601 (444h54m15.048476291s) [noise=true , address=104.36.82.4:4833 , piece_id=G7P3PPUSOU3OPMLDA4OYGR2MG2FPUAGGDNIQBO2OIW4RYR6BIQ5A, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.123325150 (444h54m15.049619838s) [noise=true , address=77.90.31.11:4558 , piece_id=XQ73S6O4ZJPVDON3MTW3UEEK6K2DIENLDAVKKYAKIQRZP5OC7KTQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.124117490 (444h54m15.050412178s) [noise=true , address=77.90.31.18:4812 , piece_id=XRMAYVPPD45JLOBJ5PJ32N222TFQE4BTTGDLYM4MF7Q7PBO64DDQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.124759188 (444h54m15.051053875s) [noise=true , address=104.36.82.19:4422 , piece_id=F3DDCJINPDSJC3LSQ6GWH7UZVP3WMBMKQ7OZTSF6A4VTOEDNUGQQ, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.125434795 (444h54m15.051729492s) [noise=true , address=45.91.95.3:4004 , piece_id=25XCJO2ZYDNRPEWZ56TBG4WPNDKU6D7VB433BCZ7VYKQEQCF4LIA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.126376895 (444h54m15.052671582s) [noise=true , address=79.127.205.233:4645 , piece_id=XCX5ACOYFQRM7YLLPFQK52HTZTJ4EJ4CPI44L4GNW2FOPNO6LBKA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.127105813 (444h54m15.053400513s) [noise=true , address=45.91.95.4:4034 , piece_id=QARMAW3PFAL4AWESAJCZY5V5AQHQ4OKEK4LG2HTD5YKIS5JZG7MA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.127854457 (444h54m15.054149149s) [noise=true , address=77.90.31.10:4549 , piece_id=EWLT62524SNFO2WFCGXKZREVVR7OPTR5JTWCDUKCA6D66S3ASF3Q, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.128593233 (444h54m15.054887929s) [noise=true , address=104.36.82.36:4624 , piece_id=AIL4TPROEY4ZFVDGL7VLHNF5NHJDWFGEAT2H5AUNMHNA5E2XOBAA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.129202345 (444h54m15.055497033s) [noise=true , address=104.36.82.22:4529 , piece_id=RPJFVUSMDAWTDEN3NYYYDFMAAW2GRMTSLYWEXL5DIV64MSEOPZLA, seg_pos={0 0}, upload=356232]: Uploading piece... Jul 11 22:05:48.230646251 (444h54m15.156940967s) [upload=356232]: Next returned nil segment Jul 11 22:05:48.230736952 (444h54m15.157031825s) [upload=356232]: Waiting for error group managing segments... Jul 11 22:05:48.428282286 (444h54m15.35457696s) [noise=true , address=77.90.31.16:4757 , piece_id=2CZYVVBJQDJUQOHSGM5J6EIZKTX53GC72MF5VKZRZO6Z5C57WIQQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:48.467746987 (444h54m15.394041674s) [noise=true , address=77.90.31.10:4549 , piece_id=EWLT62524SNFO2WFCGXKZREVVR7OPTR5JTWCDUKCA6D66S3ASF3Q, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:48.519523101 (444h54m15.445817774s) [noise=true , address=77.90.31.18:4812 , piece_id=XRMAYVPPD45JLOBJ5PJ32N222TFQE4BTTGDLYM4MF7Q7PBO64DDQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:48.619411172 (444h54m15.54570585s) [noise=true , address=45.91.116.14:4211 , piece_id=W5CAC3SOO3POEU2BVSVBGFYEP3XZQJUXKM5DQWE2KXTP3MVEU6ZA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:48.683338977 (444h54m15.609633663s) [noise=true , address=77.90.31.11:4558 , piece_id=XQ73S6O4ZJPVDON3MTW3UEEK6K2DIENLDAVKKYAKIQRZP5OC7KTQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:48.779365743 (444h54m15.705660441s) [noise=true , address=79.127.205.233:4645 , piece_id=XCX5ACOYFQRM7YLLPFQK52HTZTJ4EJ4CPI44L4GNW2FOPNO6LBKA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:48.812161844 (444h54m15.738456517s) [noise=true , address=64.49.13.154:4925 , piece_id=GDEY55ZY2HESA33OHQYLQRY2IK2NS5LOPQTCXS52KYW5QGBWYGLA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:49.094040717 (444h54m16.020335394s) [noise=true , address=77.90.31.19:4876 , piece_id=NZJZB2CJ2GJ6QYUFJC4ER3T6S22BVCRJRTQUWVYCCE7DIJKUHFWQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:49.448182353 (444h54m16.374477029s) [noise=true , address=77.90.31.15:4722 , piece_id=4NSFGP3JAPTC6PK5TQT6H3N6OWPKFJTSPTEXWAULIRQLZTBQJ5CQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:49.457437503 (444h54m16.383732197s) [noise=true , address=79.127.219.38:4034 , piece_id=ALLHZPUBKX6UJKPRB2WMBTAYVWONJEDU4YJHNXT65HRBWANF4AKA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:50.633971772 (444h54m17.560266459s) [noise=true , address=45.91.95.3:4004 , piece_id=25XCJO2ZYDNRPEWZ56TBG4WPNDKU6D7VB433BCZ7VYKQEQCF4LIA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:05:51.092276788 (444h54m18.018571476s) [noise=true , address=45.91.95.4:4034 , piece_id=QARMAW3PFAL4AWESAJCZY5V5AQHQ4OKEK4LG2HTD5YKIS5JZG7MA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: <nil> Jul 11 22:06:47.920282828 (444h55m14.846577505s) [noise=true , address=104.36.82.4:4833 , piece_id=G7P3PPUSOU3OPMLDA4OYGR2MG2FPUAGGDNIQBO2OIW4RYR6BIQ5A, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:12jfQbLaQjEXhVTbTPuawAQAbKJSN5RA8kBqz6JVDATQEGAsZ6K, address:104.36.82.4:4833): protocol: write tcp 159.223.143.39:42606->104.36.82.4:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:47.920385027 (444h55m14.846679703s) [noise=true , address=104.36.82.19:4422 , piece_id=F3DDCJINPDSJC3LSQ6GWH7UZVP3WMBMKQ7OZTSF6A4VTOEDNUGQQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1k7Bs5nYevGzsVWhp85cgDsbrpzbefborGwMAAnDsbeT9x7gio, address:104.36.82.19:4422): protocol: write tcp 159.223.143.39:49700->104.36.82.19:4422: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:47.920445144 (444h55m14.846739823s) [noise=true , address=104.36.82.36:4624 , piece_id=AIL4TPROEY4ZFVDGL7VLHNF5NHJDWFGEAT2H5AUNMHNA5E2XOBAA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1GG1c98sVuVmeJgmwXuVqxm7hKZ7Wv3LgwQGzgzGtYETe4RbF1, address:104.36.82.36:4624): protocol: write tcp 159.223.143.39:46550->104.36.82.36:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:47.920451283 (444h55m14.846745962s) [noise=true , address=104.36.82.22:4529 , piece_id=RPJFVUSMDAWTDEN3NYYYDFMAAW2GRMTSLYWEXL5DIV64MSEOPZLA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:12bNgTA4CY9ShB6Mi3QpceTEqNCr8ohCMHNVvU6S1qJp2TdVP1i, address:104.36.82.22:4529): protocol: write tcp 159.223.143.39:41026->104.36.82.22:4529: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:47.920465471 (444h55m14.846760165s) [noise=true , address=104.36.82.18:4369 , piece_id=NUCSK7GBSDCHWQ5GDGMGSPD677FGLPAG2NU7ZKQVBOR6ANAQCKFA, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:104.36.82.18:4369): protocol: write tcp 159.223.143.39:46636->104.36.82.18:4369: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:47.920475369 (444h55m14.846770048s) [noise=true , address=104.36.82.49:4011 , piece_id=6LB5ZILMX3B6CGTJFWEWGJ6GQY4LWVL6QEB5JUHP5Z324PFGPRQQ, seg_pos={0 0}, upload=356232]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: write tcp 159.223.143.39:34374->104.36.82.49:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:47.920515763 (444h55m14.846810502s) [seg_pos={0 0}, upload=356232]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:06:47.946958856 (444h55m14.873253535s) [upload=356232]: Done uploading segments... Jul 11 22:06:47.946992392 (444h55m14.873287078s) [upload=356232]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.029211407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.068881844s FAIL (2025-07-11 22:06:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.068906824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................................X..XX.XX] duration: 30.026480172s FAIL (2025-07-11 22:06:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :47.980180220 (444h54m14.906474923s) [upload=356231]: Done starting upload Jul 11 22:05:47.980393101 (444h54m14.906687774s) [upload=356231]: Uploading segments... Jul 11 22:05:47.980635324 (444h54m14.906930014s) [upload=356231]: Got next segment. Inline: false Jul 11 22:05:47.980686579 (444h54m14.906981506s) [upload=356231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:47.980692694 (444h54m14.906987387s) [upload=356231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:48.095949859 (444h54m15.022244546s) [seg_pos={0 0}, upload=356231]: Begin upload segment... Jul 11 22:05:48.096000124 (444h54m15.022294799s) [seg_pos={0 0}, upload=356231]: Done begin upload segment. Jul 11 22:05:48.096030781 (444h54m15.022325475s) [noise=true , address=104.36.82.16:4293 , piece_id=RTOSG5QCV6OZJWZMBXG7HUZOINW76E7WZRKWN5EGL2EZWZPC7SDA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.096044963 (444h54m15.022339654s) [noise=true , address=104.36.82.11:4115 , piece_id=ZPG3RN4JT4HY5CH443EXINKZJHNNGPXVII2SPI3VNTKHVTU32NZA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.096449410 (444h54m15.02274409s) [noise=true , address=45.91.116.7:4009 , piece_id=MTYPSS5U6HOV4EJBTPAXVPGOBPXVB2RVODPE2NFVLQODOPCUQXIA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.096464428 (444h54m15.02275912s) [noise=true , address=45.91.116.11:4117 , piece_id=MMJUNGT3QXKLZB2M2IZZZ3Q4FVIXYPXR3M2ZAFDRCD77TX7EQDJA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.096814022 (444h54m15.023108711s) [noise=true , address=45.91.116.4:4937 , piece_id=Y5WFTBHSDVIFKGANPTV2KO5RO6MIZUGSEDCBH7P4AFJEJFDM62TA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.097265810 (444h54m15.023560503s) [noise=true , address=104.36.82.6:4875 , piece_id=6GQQYJJMHESXF3JGISRZOCKUIHTACANFQIPKXM3RPYFSBFUW7IDA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.097631972 (444h54m15.023926668s) [noise=true , address=104.36.82.13:4185 , piece_id=KKEXDC47PRUPCSFWJGA7JOBVR3I4OESLYTCXE64EJCZVEM44BDKA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.097830028 (444h54m15.024124708s) [noise=true , address=104.36.82.16:4295 , piece_id=PZX6SPC6HYVCYLUU54XFWIC3EDNF4TYJYHND2ICXAVBXNIGLQJZA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.098088023 (444h54m15.024382721s) [noise=true , address=45.91.116.14:4191 , piece_id=OQB7QNIHXB2UFOAJN3EGWXSYJ45WCAMMY43CFD7I3PBHCUUCFBYQ, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.098617262 (444h54m15.024911958s) [noise=true , address=45.91.116.7:4011 , piece_id=EVQ6MQSGB6H3FMR6BELV67R6KZBSQU6JFKPL3PTS6U53CM5MVICQ, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.098726263 (444h54m15.025020953s) [noise=true , address=45.91.116.9:4262 , piece_id=2JUUNBSCZVTZ5VRYQJ4VESXWC2UDG2MVUHSNGHCFZX4BS4H35EZA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.099110606 (444h54m15.025405294s) [noise=true , address=104.36.82.4:4802 , piece_id=H5I7YPSONOIQJ75D6LIEV3PLOWDUVZZWWJZNE6L2PAZFK3CUKXOA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.099539974 (444h54m15.025834674s) [noise=true , address=104.36.82.6:4876 , piece_id=K7FQW6OY7XTJ5W7WBYWFQ5PEH7PXRHSX2WDAUX6FYXVXUAQ2Z7UQ, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.099907677 (444h54m15.026202367s) [noise=true , address=45.91.116.6:4975 , piece_id=NHRR54PTNJNCDVKAN4YIQFNLUEKTI2TBXUZ7YYRDVVBTXN44OXTA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.100416519 (444h54m15.02671122s) [noise=true , address=104.36.82.9:4005 , piece_id=3HU7MZ4VFIF6CZL4GPGGRYH6A6E5Y4AL7DZ5BWWLA5JQCI4V2ZGA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.100886397 (444h54m15.027181086s) [noise=true , address=45.91.116.14:4190 , piece_id=AZMQN2ECUS5T2AZSSJEDSALJACFQ3DLDZYOKWK46FBZANV7POQFA, seg_pos={0 0}, upload=356231]: Uploading piece... Jul 11 22:05:48.198354936 (444h54m15.12464963s) [upload=356231]: Next returned nil segment Jul 11 22:05:48.198375659 (444h54m15.124670355s) [upload=356231]: Waiting for error group managing segments... Jul 11 22:05:48.222725896 (444h54m15.149020573s) [noise=true , address=45.91.116.9:4262 , piece_id=2JUUNBSCZVTZ5VRYQJ4VESXWC2UDG2MVUHSNGHCFZX4BS4H35EZA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:48.245624413 (444h54m15.17191909s) [noise=true , address=45.91.116.7:4009 , piece_id=MTYPSS5U6HOV4EJBTPAXVPGOBPXVB2RVODPE2NFVLQODOPCUQXIA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:48.318695000 (444h54m15.244989688s) [noise=true , address=45.91.116.6:4975 , piece_id=NHRR54PTNJNCDVKAN4YIQFNLUEKTI2TBXUZ7YYRDVVBTXN44OXTA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:48.440263577 (444h54m15.366558255s) [noise=true , address=45.91.116.14:4191 , piece_id=OQB7QNIHXB2UFOAJN3EGWXSYJ45WCAMMY43CFD7I3PBHCUUCFBYQ, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:48.466203541 (444h54m15.392498229s) [noise=true , address=45.91.116.4:4937 , piece_id=Y5WFTBHSDVIFKGANPTV2KO5RO6MIZUGSEDCBH7P4AFJEJFDM62TA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:48.490670250 (444h54m15.416964942s) [noise=true , address=45.91.116.7:4011 , piece_id=EVQ6MQSGB6H3FMR6BELV67R6KZBSQU6JFKPL3PTS6U53CM5MVICQ, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:49.278444229 (444h54m16.204738907s) [noise=true , address=45.91.116.14:4190 , piece_id=AZMQN2ECUS5T2AZSSJEDSALJACFQ3DLDZYOKWK46FBZANV7POQFA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:05:49.288781242 (444h54m16.215075918s) [noise=true , address=45.91.116.11:4117 , piece_id=MMJUNGT3QXKLZB2M2IZZZ3Q4FVIXYPXR3M2ZAFDRCD77TX7EQDJA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: <nil> Jul 11 22:06:17.927330369 (444h54m44.853625048s) [noise=true , address=104.36.82.16:4295 , piece_id=PZX6SPC6HYVCYLUU54XFWIC3EDNF4TYJYHND2ICXAVBXNIGLQJZA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927396206 (444h54m44.853690882s) [noise=true , address=104.36.82.4:4802 , piece_id=H5I7YPSONOIQJ75D6LIEV3PLOWDUVZZWWJZNE6L2PAZFK3CUKXOA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927453876 (444h54m44.853748548s) [noise=true , address=104.36.82.6:4875 , piece_id=6GQQYJJMHESXF3JGISRZOCKUIHTACANFQIPKXM3RPYFSBFUW7IDA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927465859 (444h54m44.853760535s) [noise=true , address=104.36.82.9:4005 , piece_id=3HU7MZ4VFIF6CZL4GPGGRYH6A6E5Y4AL7DZ5BWWLA5JQCI4V2ZGA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 159.223.143.39:41650->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927517233 (444h54m44.853811906s) [noise=true , address=104.36.82.13:4185 , piece_id=KKEXDC47PRUPCSFWJGA7JOBVR3I4OESLYTCXE64EJCZVEM44BDKA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927540032 (444h54m44.853834705s) [noise=true , address=104.36.82.11:4115 , piece_id=ZPG3RN4JT4HY5CH443EXINKZJHNNGPXVII2SPI3VNTKHVTU32NZA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927590162 (444h54m44.853884837s) [noise=true , address=104.36.82.6:4876 , piece_id=K7FQW6OY7XTJ5W7WBYWFQ5PEH7PXRHSX2WDAUX6FYXVXUAQ2Z7UQ, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927601905 (444h54m44.853896577s) [noise=true , address=104.36.82.16:4293 , piece_id=RTOSG5QCV6OZJWZMBXG7HUZOINW76E7WZRKWN5EGL2EZWZPC7SDA, seg_pos={0 0}, upload=356231]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:45398->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:06:17.927621541 (444h54m44.853916231s) [seg_pos={0 0}, upload=356231]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 11 22:06:17.951654921 (444h54m44.877949597s) [upload=356231]: Done uploading segments... Jul 11 22:06:17.951678608 (444h54m44.877973297s) [upload=356231]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.02650542s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX...................................................XX..X] duration: 30.000612728s FAIL (2025-07-11 22:06:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000650728s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X..XX.X] duration: 30.02912989s FAIL (2025-07-11 22:05:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ) [upload=356221]: Done starting upload Jul 11 22:05:02.884159228 (444h53m29.810453906s) [upload=356221]: Uploading segments... Jul 11 22:05:02.884646718 (444h53m29.810941412s) [upload=356221]: Got next segment. Inline: false Jul 11 22:05:02.884735270 (444h53m29.81102996s) [upload=356221]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:05:02.884741587 (444h53m29.811036278s) [upload=356221]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:05:03.239977502 (444h53m30.16627219s) [seg_pos={0 0}, upload=356221]: Begin upload segment... Jul 11 22:05:03.240030256 (444h53m30.166324948s) [seg_pos={0 0}, upload=356221]: Done begin upload segment. Jul 11 22:05:03.240067627 (444h53m30.166362321s) [noise=true , address=77.90.31.14:4663 , piece_id=MWFG3SRAEQAGEYMHC2B4URELR4HKO5VVQFM24JIQN7Z67KRJGKXA, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.241291554 (444h53m30.167586254s) [noise=true , address=104.36.82.12:4150 , piece_id=FGAQS3RUOB2MIEWIYJBRMUVRIW6APIMV3OCUEUU3MUQXQS53IVWA, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.242004370 (444h53m30.16829907s) [noise=true , address=77.90.31.15:4697 , piece_id=H7PP2ZHUDMBA56U24LQ2MVXGJOI2WAX3GHHI23DZ4TA2WLQQA44A, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.242634105 (444h53m30.168928796s) [noise=true , address=104.36.82.8:4972 , piece_id=UQRVUK463CRITTYFSTW2O4UJCCEKWSEP5S47XFLCWMT6RNVQ7YAA, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.245326330 (444h53m30.171621028s) [noise=true , address=104.36.82.4:4801 , piece_id=BJCGIMRIWSGBIEYOWHSMF2IPVYBMBES5S4XKSK4DLYGCQSTSXDJA, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.246525149 (444h53m30.172819842s) [noise=true , address=104.36.82.6:4875 , piece_id=C7B3BDSGAJ2GHBDZVNS5TT7CG5TLN3HC5YU6D34ZXH4IT5ROFSHQ, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.247487899 (444h53m30.1737826s) [noise=true , address=45.91.116.14:4191 , piece_id=OEO4DXNMXPJ5PA76BGZ6Y3LCBFWGLH5BMXE7B4YK45ABX5IQIK7A, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.251548359 (444h53m30.177843059s) [noise=true , address=104.36.82.9:4006 , piece_id=U32GM3PACPWQSLRV2XB3QOAD7QNZO2CAJJNNTGR2RKXCPUM33AOQ, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.252162851 (444h53m30.178457548s) [noise=true , address=77.90.31.17:4770 , piece_id=GTWSZVRL3N2VG4V4HS3CWR4U267CX6G5GEIRZEM5ZU4FSL7B4BIQ, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.252700926 (444h53m30.178995674s) [noise=true , address=104.36.82.11:4114 , piece_id=WS53CGGDRW3HNRJSBBL5YK2B7E2WQ6R6BDAWMLHKFWXPK72VAQFA, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.253313824 (444h53m30.179608524s) [noise=true , address=104.36.82.10:4078 , piece_id=3CJ65VUA6KA3NOXLBCSW74O25GLJIAQOUXEVO3XYNHFEAVBRNBZQ, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.253871152 (444h53m30.180165852s) [noise=true , address=45.91.116.7:4011 , piece_id=NSKYX7IY4Q3NNGMBLKPCHRO6L5RHEGX5DL2UCP5WKILQMDO47V7Q, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.254394701 (444h53m30.180689401s) [noise=true , address=104.36.82.15:4259 , piece_id=JXXW5UAD6LUU2Y2T3PYS66Q7U67B7KLPDIBJCQRADJSOQ6ZJDSTQ, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.254917385 (444h53m30.181212096s) [noise=true , address=45.91.116.13:4153 , piece_id=BOH3CMMW6PEGIJJUESWKLVA3PB4X4TTUMQXSEQVQXPGBIIF3HW4Q, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.255493675 (444h53m30.181788377s) [noise=true , address=45.91.116.12:4227 , piece_id=5JW2BIAFBXCKADZILAZEWDPKFLSHJBSBSPLT3LUJ3RCQ6FFNJYSQ, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.256013486 (444h53m30.182308169s) [noise=true , address=104.36.82.16:4295 , piece_id=NXON7BEX4QGXQWD2UFWX2A72P3IZGVHLCP3MEWLEZQ3IAITQQW2Q, seg_pos={0 0}, upload=356221]: Uploading piece... Jul 11 22:05:03.484313136 (444h53m30.410607819s) [upload=356221]: Next returned nil segment Jul 11 22:05:03.484333264 (444h53m30.410627942s) [upload=356221]: Waiting for error group managing segments... Jul 11 22:05:03.560236060 (444h53m30.486530752s) [noise=true , address=77.90.31.14:4663 , piece_id=MWFG3SRAEQAGEYMHC2B4URELR4HKO5VVQFM24JIQN7Z67KRJGKXA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:03.741941810 (444h53m30.668236489s) [noise=true , address=77.90.31.17:4770 , piece_id=GTWSZVRL3N2VG4V4HS3CWR4U267CX6G5GEIRZEM5ZU4FSL7B4BIQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:03.742025695 (444h53m30.668320384s) [noise=true , address=77.90.31.15:4697 , piece_id=H7PP2ZHUDMBA56U24LQ2MVXGJOI2WAX3GHHI23DZ4TA2WLQQA44A, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:03.884844190 (444h53m30.811138879s) [noise=true , address=45.91.116.13:4153 , piece_id=BOH3CMMW6PEGIJJUESWKLVA3PB4X4TTUMQXSEQVQXPGBIIF3HW4Q, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:04.819858320 (444h53m31.746152994s) [noise=true , address=45.91.116.12:4227 , piece_id=5JW2BIAFBXCKADZILAZEWDPKFLSHJBSBSPLT3LUJ3RCQ6FFNJYSQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:05.336715838 (444h53m32.263010517s) [noise=true , address=45.91.116.7:4011 , piece_id=NSKYX7IY4Q3NNGMBLKPCHRO6L5RHEGX5DL2UCP5WKILQMDO47V7Q, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:06.695211707 (444h53m33.621506383s) [noise=true , address=45.91.116.14:4191 , piece_id=OEO4DXNMXPJ5PA76BGZ6Y3LCBFWGLH5BMXE7B4YK45ABX5IQIK7A, seg_pos={0 0}, upload=356221]: Done uploading piece. err: <nil> Jul 11 22:05:32.832607967 (444h53m59.758902644s) [noise=true , address=104.36.82.9:4006 , piece_id=U32GM3PACPWQSLRV2XB3QOAD7QNZO2CAJJNNTGR2RKXCPUM33AOQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.832674596 (444h53m59.758969276s) [noise=true , address=104.36.82.10:4078 , piece_id=3CJ65VUA6KA3NOXLBCSW74O25GLJIAQOUXEVO3XYNHFEAVBRNBZQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.832726736 (444h53m59.759021413s) [noise=true , address=104.36.82.15:4259 , piece_id=JXXW5UAD6LUU2Y2T3PYS66Q7U67B7KLPDIBJCQRADJSOQ6ZJDSTQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.832781191 (444h53m59.759075867s) [noise=true , address=104.36.82.16:4295 , piece_id=NXON7BEX4QGXQWD2UFWX2A72P3IZGVHLCP3MEWLEZQ3IAITQQW2Q, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.832840352 (444h53m59.759135027s) [noise=true , address=104.36.82.4:4801 , piece_id=BJCGIMRIWSGBIEYOWHSMF2IPVYBMBES5S4XKSK4DLYGCQSTSXDJA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: context deadline exceeded; context deadline exceeded Jul 11 22:05:32.832889295 (444h53m59.759183971s) [noise=true , address=104.36.82.11:4114 , piece_id=WS53CGGDRW3HNRJSBBL5YK2B7E2WQ6R6BDAWMLHKFWXPK72VAQFA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.832935105 (444h53m59.759229781s) [noise=true , address=104.36.82.12:4150 , piece_id=FGAQS3RUOB2MIEWIYJBRMUVRIW6APIMV3OCUEUU3MUQXQS53IVWA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.833099222 (444h53m59.759393899s) [noise=true , address=104.36.82.8:4972 , piece_id=UQRVUK463CRITTYFSTW2O4UJCCEKWSEP5S47XFLCWMT6RNVQ7YAA, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:05:32.833255311 (444h53m59.759549987s) [noise=true , address=104.36.82.6:4875 , piece_id=C7B3BDSGAJ2GHBDZVNS5TT7CG5TLN3HC5YU6D34ZXH4IT5ROFSHQ, seg_pos={0 0}, upload=356221]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: context deadline exceeded; context deadline exceeded Jul 11 22:05:32.833275442 (444h53m59.759570134s) [seg_pos={0 0}, upload=356221]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:05:32.858907804 (444h53m59.785202484s) [upload=356221]: Done uploading segments... Jul 11 22:05:32.858932934 (444h53m59.785227609s) [upload=356221]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.029155863s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.781951903s FAIL (2025-07-11 22:05:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.781993307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 11.324681003s FAIL (2025-07-11 22:05:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.324708082s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.920288374s FAIL (2025-07-11 22:04:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 18.920314133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X..XX.] duration: 1.384232907s FAIL (2025-07-11 22:04:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 2QPIZYKR3B7QFCIP7OPLO42FJGYWTKM5Q, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.035991091 (444h52m5.962285777s) [noise=true , address=104.36.82.8:4969 , piece_id=OR4TF665LMGJ5V5C2BKVIRUP2IHFIA5CIWEKDP6KHFMX32XSECFA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036058751 (444h52m5.962353446s) [noise=true , address=77.90.31.11:4553 , piece_id=646RYZB4QSV5U45RGUUNYKIEHHVYH6JSWJEVYPYMW4L6SL5Q3XJQ, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036362206 (444h52m5.962656892s) [noise=true , address=104.36.82.2:4729 , piece_id=QONL5VROA35M2TUG3PVBXBEBALYU3N5BHLJ2M4ZYLW2XAAFD4OLQ, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036436321 (444h52m5.962730994s) [noise=true , address=45.91.116.7:4011 , piece_id=G5SREX742C6XKU2ZUCUHVW2JF3LQUNJGCFMYPWITXGPKHGFX4LNQ, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036748771 (444h52m5.963043458s) [noise=true , address=104.36.82.9:4007 , piece_id=F6BSFPMKVGGKEARAHPVAPNVG53ZUC6O7F3HEKCWI4LTCE7Y2TNVA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.037294718 (444h52m5.963589412s) [noise=true , address=77.90.31.12:4589 , piece_id=QXGMCVPLCLTOXUD7ZFLVFTU7ZFL4LGREAEHPSY7BGIYYA4SC5YYA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.039439542 (444h52m5.965734238s) [noise=true , address=104.36.82.10:4079 , piece_id=BTQVZWSIMDRRA5STDS5NNJM3P2B5MNSXNJTXXONTHV7CYDYNJ5TA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.126299330 (444h52m6.052594024s) [upload=356202]: Next returned nil segment Jul 11 22:03:39.126317798 (444h52m6.052612497s) [upload=356202]: Waiting for error group managing segments... Jul 11 22:03:39.132400117 (444h52m6.058694797s) [noise=true , address=77.90.31.10:4520 , piece_id=FMLEVIT54PA4BIXX5OMZRSZJ7FF5HF4LOF33UXKDVKM7BNVQI7EA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.135106726 (444h52m6.061401413s) [noise=true , address=45.91.116.7:4011 , piece_id=G5SREX742C6XKU2ZUCUHVW2JF3LQUNJGCFMYPWITXGPKHGFX4LNQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.142872818 (444h52m6.069167507s) [noise=true , address=77.90.31.12:4589 , piece_id=QXGMCVPLCLTOXUD7ZFLVFTU7ZFL4LGREAEHPSY7BGIYYA4SC5YYA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.319001106 (444h52m6.245295794s) [noise=true , address=77.90.31.11:4553 , piece_id=646RYZB4QSV5U45RGUUNYKIEHHVYH6JSWJEVYPYMW4L6SL5Q3XJQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.520601428 (444h52m6.446896113s) [noise=true , address=77.90.31.19:4842 , piece_id=VS774W3L5R7FG73IFS3HN43A5VCMHJSUFHCB7UPZ7QMHNKDO6MJQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:04:08.975421791 (444h52m35.901716478s) [noise=true , address=104.36.82.4:4804 , piece_id=KZCGLLVQQQINJ7VBAACKZ2KZNWNJSHVCPO5MVRCHFDL4RVEV3Y5Q, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:53316->104.36.82.4:4804: use of closed network connection; write tcp 159.223.143.39:53316->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53316->104.36.82.4:4804: use of closed network connection Jul 11 22:04:08.975507931 (444h52m35.901802619s) [noise=true , address=104.36.82.14:4221 , piece_id=G7BWMZYTURUWCBISWA22QPIZYKR3B7QFCIP7OPLO42FJGYWTKM5Q, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:41824->104.36.82.14:4221: use of closed network connection; write tcp 159.223.143.39:41824->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41824->104.36.82.14:4221: use of closed network connection Jul 11 22:04:08.975570475 (444h52m35.901865162s) [noise=true , address=104.36.82.10:4079 , piece_id=BTQVZWSIMDRRA5STDS5NNJM3P2B5MNSXNJTXXONTHV7CYDYNJ5TA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:59732->104.36.82.10:4079: use of closed network connection; write tcp 159.223.143.39:59732->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59732->104.36.82.10:4079: use of closed network connection Jul 11 22:04:08.975629934 (444h52m35.901924623s) [noise=true , address=104.36.82.6:4876 , piece_id=22HOIKYJ4XWDWVWG2JQJYPEUS2XONOVG6OQT4QX5WV5HXDYR2WHQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:59290->104.36.82.6:4876: use of closed network connection; write tcp 159.223.143.39:59290->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59290->104.36.82.6:4876: use of closed network connection Jul 11 22:04:08.975650317 (444h52m35.901945005s) [noise=true , address=104.36.82.5:4840 , piece_id=EUW3WRNN4SEIYDNPGE4LIQZEAXGXIQUPIKFRUOZVBAPNZGBK5HKQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:38360->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:38360->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38360->104.36.82.5:4840: use of closed network connection Jul 11 22:04:08.975694898 (444h52m35.901989587s) [noise=true , address=104.36.82.8:4969 , piece_id=OR4TF665LMGJ5V5C2BKVIRUP2IHFIA5CIWEKDP6KHFMX32XSECFA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:40366->104.36.82.8:4969: use of closed network connection; write tcp 159.223.143.39:40366->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:40366->104.36.82.8:4969: use of closed network connection Jul 11 22:04:08.975742393 (444h52m35.902037082s) [noise=true , address=104.36.82.14:4223 , piece_id=S2YITTQO4XI4ELDWXPFPXMZBU3A5VQE7YFQPQVCXMVKQME34VQEQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:40112->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:40112->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:40112->104.36.82.14:4223: use of closed network connection Jul 11 22:04:08.975795557 (444h52m35.902090242s) [noise=true , address=104.36.82.7:4912 , piece_id=Y4BVCPSID5B3MCU5JUEPN7OD7KDTYYWJXAZZHVBBGP7HKTN77FTA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:52036->104.36.82.7:4912: use of closed network connection; write tcp 159.223.143.39:52036->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:52036->104.36.82.7:4912: use of closed network connection Jul 11 22:04:08.975827742 (444h52m35.902122428s) [noise=true , address=104.36.82.9:4007 , piece_id=F6BSFPMKVGGKEARAHPVAPNVG53ZUC6O7F3HEKCWI4LTCE7Y2TNVA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:57508->104.36.82.9:4007: use of closed network connection; write tcp 159.223.143.39:57508->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57508->104.36.82.9:4007: use of closed network connection Jul 11 22:04:08.975838582 (444h52m35.902133262s) [noise=true , address=104.36.82.2:4729 , piece_id=QONL5VROA35M2TUG3PVBXBEBALYU3N5BHLJ2M4ZYLW2XAAFD4OLQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51648->104.36.82.2:4729: use of closed network connection; write tcp 159.223.143.39:51648->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51648->104.36.82.2:4729: use of closed network connection Jul 11 22:04:08.975855304 (444h52m35.90214999s) [noise=true , address=104.36.82.13:4185 , piece_id=R4L2UW2YG4YO52JST46FFTLF7PQABSS424TVS3UZKUU2XT6HY5KA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:33728->104.36.82.13:4185: use of closed network connection; write tcp 159.223.143.39:33728->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33728->104.36.82.13:4185: use of closed network connection Jul 11 22:04:08.975873741 (444h52m35.902168431s) [seg_pos={0 0}, upload=356202]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 11 22:04:09.001500740 (444h52m35.927795466s) [upload=356202]: Done uploading segments... Jul 11 22:04:09.001550602 (444h52m35.927845307s) [upload=356202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 1.384258878s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XX...................................................XX..] duration: 823.721792ms FAIL (2025-07-11 22:04:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 823.748859ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X..XX] duration: 30.028467519s FAIL (2025-07-11 22:04:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 2QPIZYKR3B7QFCIP7OPLO42FJGYWTKM5Q, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.035991091 (444h52m5.962285777s) [noise=true , address=104.36.82.8:4969 , piece_id=OR4TF665LMGJ5V5C2BKVIRUP2IHFIA5CIWEKDP6KHFMX32XSECFA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036058751 (444h52m5.962353446s) [noise=true , address=77.90.31.11:4553 , piece_id=646RYZB4QSV5U45RGUUNYKIEHHVYH6JSWJEVYPYMW4L6SL5Q3XJQ, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036362206 (444h52m5.962656892s) [noise=true , address=104.36.82.2:4729 , piece_id=QONL5VROA35M2TUG3PVBXBEBALYU3N5BHLJ2M4ZYLW2XAAFD4OLQ, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036436321 (444h52m5.962730994s) [noise=true , address=45.91.116.7:4011 , piece_id=G5SREX742C6XKU2ZUCUHVW2JF3LQUNJGCFMYPWITXGPKHGFX4LNQ, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.036748771 (444h52m5.963043458s) [noise=true , address=104.36.82.9:4007 , piece_id=F6BSFPMKVGGKEARAHPVAPNVG53ZUC6O7F3HEKCWI4LTCE7Y2TNVA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.037294718 (444h52m5.963589412s) [noise=true , address=77.90.31.12:4589 , piece_id=QXGMCVPLCLTOXUD7ZFLVFTU7ZFL4LGREAEHPSY7BGIYYA4SC5YYA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.039439542 (444h52m5.965734238s) [noise=true , address=104.36.82.10:4079 , piece_id=BTQVZWSIMDRRA5STDS5NNJM3P2B5MNSXNJTXXONTHV7CYDYNJ5TA, seg_pos={0 0}, upload=356202]: Uploading piece... Jul 11 22:03:39.126299330 (444h52m6.052594024s) [upload=356202]: Next returned nil segment Jul 11 22:03:39.126317798 (444h52m6.052612497s) [upload=356202]: Waiting for error group managing segments... Jul 11 22:03:39.132400117 (444h52m6.058694797s) [noise=true , address=77.90.31.10:4520 , piece_id=FMLEVIT54PA4BIXX5OMZRSZJ7FF5HF4LOF33UXKDVKM7BNVQI7EA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.135106726 (444h52m6.061401413s) [noise=true , address=45.91.116.7:4011 , piece_id=G5SREX742C6XKU2ZUCUHVW2JF3LQUNJGCFMYPWITXGPKHGFX4LNQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.142872818 (444h52m6.069167507s) [noise=true , address=77.90.31.12:4589 , piece_id=QXGMCVPLCLTOXUD7ZFLVFTU7ZFL4LGREAEHPSY7BGIYYA4SC5YYA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.319001106 (444h52m6.245295794s) [noise=true , address=77.90.31.11:4553 , piece_id=646RYZB4QSV5U45RGUUNYKIEHHVYH6JSWJEVYPYMW4L6SL5Q3XJQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:03:39.520601428 (444h52m6.446896113s) [noise=true , address=77.90.31.19:4842 , piece_id=VS774W3L5R7FG73IFS3HN43A5VCMHJSUFHCB7UPZ7QMHNKDO6MJQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: <nil> Jul 11 22:04:08.975421791 (444h52m35.901716478s) [noise=true , address=104.36.82.4:4804 , piece_id=KZCGLLVQQQINJ7VBAACKZ2KZNWNJSHVCPO5MVRCHFDL4RVEV3Y5Q, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:53316->104.36.82.4:4804: use of closed network connection; write tcp 159.223.143.39:53316->104.36.82.4:4804: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53316->104.36.82.4:4804: use of closed network connection Jul 11 22:04:08.975507931 (444h52m35.901802619s) [noise=true , address=104.36.82.14:4221 , piece_id=G7BWMZYTURUWCBISWA22QPIZYKR3B7QFCIP7OPLO42FJGYWTKM5Q, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:41824->104.36.82.14:4221: use of closed network connection; write tcp 159.223.143.39:41824->104.36.82.14:4221: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:41824->104.36.82.14:4221: use of closed network connection Jul 11 22:04:08.975570475 (444h52m35.901865162s) [noise=true , address=104.36.82.10:4079 , piece_id=BTQVZWSIMDRRA5STDS5NNJM3P2B5MNSXNJTXXONTHV7CYDYNJ5TA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 159.223.143.39:59732->104.36.82.10:4079: use of closed network connection; write tcp 159.223.143.39:59732->104.36.82.10:4079: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59732->104.36.82.10:4079: use of closed network connection Jul 11 22:04:08.975629934 (444h52m35.901924623s) [noise=true , address=104.36.82.6:4876 , piece_id=22HOIKYJ4XWDWVWG2JQJYPEUS2XONOVG6OQT4QX5WV5HXDYR2WHQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:59290->104.36.82.6:4876: use of closed network connection; write tcp 159.223.143.39:59290->104.36.82.6:4876: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:59290->104.36.82.6:4876: use of closed network connection Jul 11 22:04:08.975650317 (444h52m35.901945005s) [noise=true , address=104.36.82.5:4840 , piece_id=EUW3WRNN4SEIYDNPGE4LIQZEAXGXIQUPIKFRUOZVBAPNZGBK5HKQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:38360->104.36.82.5:4840: use of closed network connection; write tcp 159.223.143.39:38360->104.36.82.5:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38360->104.36.82.5:4840: use of closed network connection Jul 11 22:04:08.975694898 (444h52m35.901989587s) [noise=true , address=104.36.82.8:4969 , piece_id=OR4TF665LMGJ5V5C2BKVIRUP2IHFIA5CIWEKDP6KHFMX32XSECFA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 159.223.143.39:40366->104.36.82.8:4969: use of closed network connection; write tcp 159.223.143.39:40366->104.36.82.8:4969: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:40366->104.36.82.8:4969: use of closed network connection Jul 11 22:04:08.975742393 (444h52m35.902037082s) [noise=true , address=104.36.82.14:4223 , piece_id=S2YITTQO4XI4ELDWXPFPXMZBU3A5VQE7YFQPQVCXMVKQME34VQEQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:40112->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:40112->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:40112->104.36.82.14:4223: use of closed network connection Jul 11 22:04:08.975795557 (444h52m35.902090242s) [noise=true , address=104.36.82.7:4912 , piece_id=Y4BVCPSID5B3MCU5JUEPN7OD7KDTYYWJXAZZHVBBGP7HKTN77FTA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:52036->104.36.82.7:4912: use of closed network connection; write tcp 159.223.143.39:52036->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:52036->104.36.82.7:4912: use of closed network connection Jul 11 22:04:08.975827742 (444h52m35.902122428s) [noise=true , address=104.36.82.9:4007 , piece_id=F6BSFPMKVGGKEARAHPVAPNVG53ZUC6O7F3HEKCWI4LTCE7Y2TNVA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:57508->104.36.82.9:4007: use of closed network connection; write tcp 159.223.143.39:57508->104.36.82.9:4007: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57508->104.36.82.9:4007: use of closed network connection Jul 11 22:04:08.975838582 (444h52m35.902133262s) [noise=true , address=104.36.82.2:4729 , piece_id=QONL5VROA35M2TUG3PVBXBEBALYU3N5BHLJ2M4ZYLW2XAAFD4OLQ, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 159.223.143.39:51648->104.36.82.2:4729: use of closed network connection; write tcp 159.223.143.39:51648->104.36.82.2:4729: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51648->104.36.82.2:4729: use of closed network connection Jul 11 22:04:08.975855304 (444h52m35.90214999s) [noise=true , address=104.36.82.13:4185 , piece_id=R4L2UW2YG4YO52JST46FFTLF7PQABSS424TVS3UZKUU2XT6HY5KA, seg_pos={0 0}, upload=356202]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 159.223.143.39:33728->104.36.82.13:4185: use of closed network connection; write tcp 159.223.143.39:33728->104.36.82.13:4185: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33728->104.36.82.13:4185: use of closed network connection Jul 11 22:04:08.975873741 (444h52m35.902168431s) [seg_pos={0 0}, upload=356202]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 11 22:04:09.001500740 (444h52m35.927795466s) [upload=356202]: Done uploading segments... Jul 11 22:04:09.001550602 (444h52m35.927845307s) [upload=356202]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.02849057s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XX...................................................XX.] duration: 3.402610942s FAIL (2025-07-11 22:04:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402672514s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.6484897s FAIL (2025-07-11 22:03:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.648516017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XX...................................................XX] duration: 30.000585808s FAIL (2025-07-11 22:03:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000603979s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.05340843s FAIL (2025-07-11 22:03:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ) [noise=true , address=104.36.82.16:4293 , piece_id=W3XTDZNAC7IRSIRPGPIHWHVZFQVVRYYRDQCSOO42SWM2N4PKMFYQ, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.931616055 (444h51m35.857910732s) [noise=true , address=104.36.82.7:4912 , piece_id=RFRXB7YY4WRMH2B6TQTSR5CDCSYGSZQDBNNITMUVAT2NFDSUJO2Q, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.931733003 (444h51m35.858027678s) [noise=true , address=104.36.82.10:4077 , piece_id=BFRSB74NB6ELQBRVMCLUXKGU4WAYQQIDTC3HOBKM5PV4CACTFMRQ, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.932034304 (444h51m35.858329003s) [noise=true , address=104.36.82.9:4006 , piece_id=77NIIKUZXNLVLIVIAON6VLBND4DTMWL2CINFIUG6DZKRCMU2MHDQ, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.932197594 (444h51m35.858492271s) [noise=true , address=104.36.82.10:4078 , piece_id=ET2UWWZKZ5MPN7TWR63I5J6PRXEJQYXVBABCXUX4CJG6M3LI63WQ, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.932517673 (444h51m35.858812371s) [noise=true , address=104.36.82.6:4874 , piece_id=K2J4LDN3IRHZV2KXDWPZONDR3JN25RMF3DHNPEGBLVIGEAFHN4XQ, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.932633185 (444h51m35.858927875s) [noise=true , address=45.91.116.6:4974 , piece_id=LBB5BYGI6XT4QCJRYTXEBY4Y74NDA3O42NO6XRWI2GBQOIZXRKPQ, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.933139870 (444h51m35.859434563s) [noise=true , address=45.91.116.12:4225 , piece_id=5PPLJY73VZO6JNL7OTGZAPQDRD4TNBAKOJQUSD4GHCKRD63XBXDA, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.933638600 (444h51m35.859933309s) [noise=true , address=45.91.116.9:4261 , piece_id=TG3CGLXG5QMCMSGERENVHW5EXF2PC6PBIGQFU7GLIF6DRMGUWM5A, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.934063666 (444h51m35.860358363s) [noise=true , address=104.36.82.13:4186 , piece_id=AJDB477DPPTVJOTZASFEHWCZ72PC6F74CG2BAK6FYGTDR2QP4TDA, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.934514043 (444h51m35.86080874s) [noise=true , address=104.36.82.14:4223 , piece_id=PLPVWFLAOBCDDGPO6KXYRXPB6W3ZSFCUO62VWGJBBCXXXJZNCBMA, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.936404071 (444h51m35.86269877s) [noise=true , address=104.36.82.16:4295 , piece_id=PGIJX5E7QXPJLLE24632B5PTW2ZCMX5JBPPSXRK6TT3K3PB7DMQA, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.936904265 (444h51m35.863198958s) [noise=true , address=77.90.31.14:4661 , piece_id=U53O7ZXIEGVE7CH6DPICFXNSJPIULIYEKCHP7ZPRFXTIPX6UWZPA, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.937378117 (444h51m35.863672811s) [noise=true , address=45.91.116.11:4117 , piece_id=Y3NFAVLFEKW252DZKYRLJPZYRKR4SGMABARWBMJERDL5X2F64GMA, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:08.938795545 (444h51m35.865090241s) [noise=true , address=45.91.116.7:4010 , piece_id=3AGQNJP6HR2TPTAAXYBEFKLRKF6XUJWYCOCALJHV6J7XJ4Z2CH7A, seg_pos={0 0}, upload=356195]: Uploading piece... Jul 11 22:03:09.053313062 (444h51m35.979607736s) [upload=356195]: Next returned nil segment Jul 11 22:03:09.053336889 (444h51m35.979631577s) [upload=356195]: Waiting for error group managing segments... Jul 11 22:03:09.062146245 (444h51m35.988440923s) [noise=true , address=45.91.116.6:4974 , piece_id=LBB5BYGI6XT4QCJRYTXEBY4Y74NDA3O42NO6XRWI2GBQOIZXRKPQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:09.063948287 (444h51m35.990242976s) [noise=true , address=77.90.31.14:4661 , piece_id=U53O7ZXIEGVE7CH6DPICFXNSJPIULIYEKCHP7ZPRFXTIPX6UWZPA, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:09.076091837 (444h51m36.002386514s) [noise=true , address=45.91.116.11:4117 , piece_id=Y3NFAVLFEKW252DZKYRLJPZYRKR4SGMABARWBMJERDL5X2F64GMA, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:09.081743958 (444h51m36.008038657s) [noise=true , address=45.91.116.8:4048 , piece_id=NMFSQ6ARGGTGX7HATIC2J5S62U53K7SEA7YIFYNOEFH7FCR2O5ZQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:09.124730535 (444h51m36.051025216s) [noise=true , address=45.91.116.7:4010 , piece_id=3AGQNJP6HR2TPTAAXYBEFKLRKF6XUJWYCOCALJHV6J7XJ4Z2CH7A, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:09.208811129 (444h51m36.135105809s) [noise=true , address=45.91.116.9:4261 , piece_id=TG3CGLXG5QMCMSGERENVHW5EXF2PC6PBIGQFU7GLIF6DRMGUWM5A, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:10.208272258 (444h51m37.134566953s) [noise=true , address=45.91.116.12:4225 , piece_id=5PPLJY73VZO6JNL7OTGZAPQDRD4TNBAKOJQUSD4GHCKRD63XBXDA, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:28.673386019 (444h51m55.599680692s) [noise=true , address=104.36.82.10:4078 , piece_id=ET2UWWZKZ5MPN7TWR63I5J6PRXEJQYXVBABCXUX4CJG6M3LI63WQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: <nil> Jul 11 22:03:38.844341134 (444h52m5.770635807s) [noise=true , address=104.36.82.13:4186 , piece_id=AJDB477DPPTVJOTZASFEHWCZ72PC6F74CG2BAK6FYGTDR2QP4TDA, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:45122->104.36.82.13:4186: use of closed network connection; write tcp 159.223.143.39:45122->104.36.82.13:4186: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:45122->104.36.82.13:4186: use of closed network connection Jul 11 22:03:38.844402060 (444h52m5.770696732s) [noise=true , address=104.36.82.16:4293 , piece_id=W3XTDZNAC7IRSIRPGPIHWHVZFQVVRYYRDQCSOO42SWM2N4PKMFYQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:49950->104.36.82.16:4293: use of closed network connection; write tcp 159.223.143.39:49950->104.36.82.16:4293: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49950->104.36.82.16:4293: use of closed network connection Jul 11 22:03:38.844450097 (444h52m5.770744783s) [noise=true , address=104.36.82.9:4006 , piece_id=77NIIKUZXNLVLIVIAON6VLBND4DTMWL2CINFIUG6DZKRCMU2MHDQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:60764->104.36.82.9:4006: use of closed network connection; write tcp 159.223.143.39:60764->104.36.82.9:4006: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:60764->104.36.82.9:4006: use of closed network connection Jul 11 22:03:38.844454062 (444h52m5.770748752s) [noise=true , address=104.36.82.7:4912 , piece_id=RFRXB7YY4WRMH2B6TQTSR5CDCSYGSZQDBNNITMUVAT2NFDSUJO2Q, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:37406->104.36.82.7:4912: use of closed network connection; write tcp 159.223.143.39:37406->104.36.82.7:4912: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:37406->104.36.82.7:4912: use of closed network connection Jul 11 22:03:38.844487375 (444h52m5.770782047s) [noise=true , address=104.36.82.16:4295 , piece_id=PGIJX5E7QXPJLLE24632B5PTW2ZCMX5JBPPSXRK6TT3K3PB7DMQA, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:49386->104.36.82.16:4295: use of closed network connection; context deadline exceeded Jul 11 22:03:38.844520001 (444h52m5.770814673s) [noise=true , address=104.36.82.14:4223 , piece_id=PLPVWFLAOBCDDGPO6KXYRXPB6W3ZSFCUO62VWGJBBCXXXJZNCBMA, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:38354->104.36.82.14:4223: use of closed network connection; write tcp 159.223.143.39:38354->104.36.82.14:4223: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:38354->104.36.82.14:4223: use of closed network connection Jul 11 22:03:38.844593373 (444h52m5.770888045s) [noise=true , address=104.36.82.6:4874 , piece_id=K2J4LDN3IRHZV2KXDWPZONDR3JN25RMF3DHNPEGBLVIGEAFHN4XQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:46626->104.36.82.6:4874: use of closed network connection; write tcp 159.223.143.39:46626->104.36.82.6:4874: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:46626->104.36.82.6:4874: use of closed network connection Jul 11 22:03:38.844694874 (444h52m5.77098955s) [noise=true , address=104.36.82.10:4077 , piece_id=BFRSB74NB6ELQBRVMCLUXKGU4WAYQQIDTC3HOBKM5PV4CACTFMRQ, seg_pos={0 0}, upload=356195]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: read tcp 159.223.143.39:58674->104.36.82.10:4077: use of closed network connection; context deadline exceeded Jul 11 22:03:38.844736332 (444h52m5.771031021s) [seg_pos={0 0}, upload=356195]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 11 22:03:38.896086898 (444h52m5.822381586s) [upload=356195]: Done uploading segments... Jul 11 22:03:38.896117069 (444h52m5.822411753s) [upload=356195]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.05345064s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.09515495s FAIL (2025-07-11 22:03:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.095174169s core uplink us-select-1 us1-placement-20

1
...
3081
3082
3083
3084
3085
...
3308