Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 149946 reports (Page 2772 of 2999)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X..XX.XXXXX...................................] duration: 579.018076ms FAIL (2025-07-11 22:33:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 579.05169ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 677.795839ms FAIL (2025-07-11 22:33:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 677.822046ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................XX..XXXX.................................] duration: 540.166508ms FAIL (2025-07-11 22:33:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 540.18038ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X..XX.XXXXX..................................] duration: 462.694441ms FAIL (2025-07-11 22:33:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 462.737535ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 787.31945ms FAIL (2025-07-11 22:32:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 787.349152ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 684.916759ms FAIL (2025-07-11 22:32:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 684.943381ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X..XX.XXXXX.................................] duration: 608.722132ms FAIL (2025-07-11 22:32:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 608.762637ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................XX..XXXX................................] duration: 3.843928299s FAIL (2025-07-11 22:32:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.844022932s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 593.576865ms FAIL (2025-07-11 22:32:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 593.603272ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................XX..XXXX...............................] duration: 1.31580854s FAIL (2025-07-11 22:32:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.315841494s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 704.35827ms FAIL (2025-07-11 22:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 704.386689ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X..XX.XXXXX................................] duration: 1.271847152s FAIL (2025-07-11 22:31:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.271908388s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.286813435s FAIL (2025-07-11 22:31:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.286840736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 763.797947ms FAIL (2025-07-11 22:31:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 763.829814ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................XX..XXXX..............................] duration: 486.865707ms FAIL (2025-07-11 22:31:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.892184ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 683.692738ms FAIL (2025-07-11 22:31:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 683.724723ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X..XX.XXXXX...............................] duration: 486.332287ms FAIL (2025-07-11 22:30:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 486.365246ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 3.402435763s FAIL (2025-07-11 22:30:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 3.402469959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 817.361146ms FAIL (2025-07-11 22:30:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 817.388599ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................XX..XXXX.............................] duration: 786.979866ms FAIL (2025-07-11 22:30:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 787.008154ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X..XX.XXXXX..............................] duration: 1.009086362s FAIL (2025-07-11 22:29:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.009115004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 1.628900646s FAIL (2025-07-11 22:29:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1.628929243s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................XX..XXXX............................] duration: 511.465403ms FAIL (2025-07-11 22:29:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.499631ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 674.129663ms FAIL (2025-07-11 22:29:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 674.16195ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................XX..XXXX...........................] duration: 586.786181ms FAIL (2025-07-11 22:29:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 586.808435ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 641.289554ms FAIL (2025-07-11 22:29:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 641.442831ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 759.876848ms FAIL (2025-07-11 22:28:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 760.102455ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X..XX.XXXXX.............................] duration: 461.574427ms FAIL (2025-07-11 22:28:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 461.607521ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X..XX.XXXXX............................] duration: 558.11239ms FAIL (2025-07-11 22:28:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 558.150911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 736.415252ms FAIL (2025-07-11 22:28:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 736.458203ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 658.825754ms FAIL (2025-07-11 22:28:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 658.854037ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................XX..XXXX..........................] duration: 667.980182ms FAIL (2025-07-11 22:28:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 668.025973ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X..XX.XXXXX...........................] duration: 496.585923ms FAIL (2025-07-11 22:27:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 496.61516ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 657.388605ms FAIL (2025-07-11 22:27:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 658.131219ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 1.090921008s FAIL (2025-07-11 22:27:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1.090949152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X..XX.XXXXX..........................] duration: 554.413575ms FAIL (2025-07-11 22:27:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 554.445106ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................XX..XXXX.........................] duration: 518.308702ms FAIL (2025-07-11 22:27:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 518.331284ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 609.477832ms FAIL (2025-07-11 22:26:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 609.513135ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 681.989872ms FAIL (2025-07-11 22:26:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 682.04727ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................XX..XXXX........................] duration: 551.804643ms FAIL (2025-07-11 22:26:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 551.840674ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X..XX.XXXXX.........................] duration: 535.22045ms FAIL (2025-07-11 22:26:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 535.257127ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 610.412485ms FAIL (2025-07-11 22:25:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 610.44792ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XX..XXXX.......................] duration: 568.312947ms FAIL (2025-07-11 22:25:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 568.37219ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 678.424384ms FAIL (2025-07-11 22:25:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 678.454989ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................X..XX.XXXXX........................] duration: 500.422053ms FAIL (2025-07-11 22:25:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 500.458173ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................X..XX.XXXXX.......................] duration: 481.319177ms FAIL (2025-07-11 22:24:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 481.349945ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 696.52028ms FAIL (2025-07-11 22:24:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 696.547674ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XX..XXXX......................] duration: 507.643247ms FAIL (2025-07-11 22:24:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 507.672307ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 720.568257ms FAIL (2025-07-11 22:24:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 866s) [upload=356245]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:06:48.119568664 (444h55m15.04586334s) [seg_pos={0 0}, upload=356245]: Begin upload segment... Jul 11 22:06:48.119615842 (444h55m15.045910522s) [seg_pos={0 0}, upload=356245]: Done begin upload segment. Jul 11 22:06:48.119645440 (444h55m15.045940129s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.119749710 (444h55m15.046044401s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120044923 (444h55m15.046339624s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120425220 (444h55m15.046719909s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120536757 (444h55m15.046831453s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.120781035 (444h55m15.047075725s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121218268 (444h55m15.04751296s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121366724 (444h55m15.047661422s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121558593 (444h55m15.047853282s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.121926326 (444h55m15.048221081s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122049755 (444h55m15.048344451s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122244223 (444h55m15.048538918s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122608074 (444h55m15.048902763s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122632995 (444h55m15.048927693s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.122942478 (444h55m15.04923717s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123230436 (444h55m15.049525147s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.123750704 (444h55m15.050045414s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.124199928 (444h55m15.05049464s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Uploading piece... Jul 11 22:06:48.186620753 (444h55m15.112915429s) [upload=356245]: Next returned nil segment Jul 11 22:06:48.186635352 (444h55m15.112930034s) [upload=356245]: Waiting for error group managing segments... Jul 11 22:06:48.202636026 (444h55m15.128930713s) [noise=true , address=79.127.219.38:4007 , piece_id=LXBTPXTLAFZEBMXPLYG2UKCUJQDUH3KJCPCA4ZMNG2AGDQ2UAZIA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.234091255 (444h55m15.160385931s) [noise=true , address=45.91.116.6:4979 , piece_id=HVABE3YRW33375ANQC3EQNP2IPDBNNBROSL45E5Z3JAGF7FI6AMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.414926167 (444h55m15.34122084s) [noise=true , address=156.146.43.225:4441 , piece_id=RTSGRV4W7C7GYYU576IFIQJFZGVNHSXRGOGHNWWDDFECVX7CABWQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.625630702 (444h55m15.551925379s) [noise=true , address=77.90.31.10:4532 , piece_id=MWGECY3Q2D7HWSGBT35GN32CSKBBTLPX7T6YVM35WG264QR5O7GQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.644694298 (444h55m15.570988978s) [noise=true , address=77.90.31.19:4853 , piece_id=6IPUK5RV3MMHSXB7YNWUTSEJ22LVFZMERVCUUJSVCS2LBXN5RDPA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.661223922 (444h55m15.587518616s) [noise=true , address=77.90.31.11:4563 , piece_id=UYVTVTSIA2POTGSAOMX72Y7LMH6PEPXVJKDU2NTVETGCL7HS3XKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.752446757 (444h55m15.678741438s) [noise=true , address=77.90.31.16:4764 , piece_id=3QSDUTYTD5GT4DEGSWFOQ4GVIEF3OS4TRL3AQQUZATKI5YSAXLHA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:48.802487487 (444h55m15.728782163s) [noise=true , address=64.49.13.154:4936 , piece_id=TIPDXA2CUTEYAL25BNKTPLLVL6GB4NBXTLJBLR7FUQHHFTZVHBKQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:50.847786353 (444h55m17.774081045s) [noise=true , address=45.91.95.3:4013 , piece_id=2SA54WE7I22KEBLLATRKXMONZKSOWBVKOH3OOJGKOC6EQVVIUDQQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:52.924306132 (444h55m19.850600811s) [noise=true , address=77.90.31.14:4667 , piece_id=UORNPVGDB4EO23WZ7QDE2RXZNMAFPUUHQACHLWH6JLYWTBOR3CVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:06:59.911842283 (444h55m26.838136959s) [noise=true , address=77.90.31.12:4617 , piece_id=J5BERU4B5CCTZENA2X5VH7OR6SXM5LLYMWYODBHN57OQ2ZRZSFWA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:03.225901671 (444h55m30.152196358s) [noise=true , address=77.90.31.13:4636 , piece_id=BGE3LTGWLFWHP5YTK7DAVSH34VV76O4NDOUAMKLQXGMRNNJBKDMA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: <nil> Jul 11 22:07:48.027353535 (444h56m14.953648215s) [noise=true , address=104.36.82.49:4979 , piece_id=YSA4NZTOYU2VNDBCNYZFI4DFGSSBISWVOQIUXIQMBLMPS3CVASLA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027419215 (444h56m14.953713896s) [noise=true , address=104.36.82.17:4358 , piece_id=FHIHJDGJZAO6GDH7VW25VY2VVUPKCPZ5RSOKCP4RHIOIKQQOFYLQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1FyLtRsiMJkXiZmQpvsVwUDUt3TunVJ5BpN61eeNqXddJbQGnK, address:104.36.82.17:4358): protocol: write tcp 159.223.143.39:59402->104.36.82.17:4358: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027725812 (444h56m14.954020488s) [noise=true , address=104.36.82.19:4409 , piece_id=VZ3NZTIU2WBPIOGDYGVN4S7PL5BQGJ7N2ZHTD2K5JTFDZZYXZ5IQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:125CLa5WEhb2C5y9CJtyy38JfqhUaWusA4Vr4FPdkXMFoMQ7SiP, address:104.36.82.19:4409): protocol: write tcp 159.223.143.39:59260->104.36.82.19:4409: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027828909 (444h56m14.954123597s) [noise=true , address=104.36.82.51:4059 , piece_id=XODFR4WFJO3JKOP75STMOM6U7PZ7WBNILRF2TFZFNMOFFGB5JQZQ, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: rpc: dial tcp 104.36.82.51:4059: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027844777 (444h56m14.954139466s) [noise=true , address=104.36.82.19:4427 , piece_id=KPOZV6WYFGOZ4AXCUSLXP6ENCFJBJNM2XDWFRCCRCEJ6Z6V4VRXA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG, address:104.36.82.19:4427): protocol: rpc: dial tcp 104.36.82.19:4427: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027906529 (444h56m14.954201217s) [noise=true , address=104.36.82.35:4615 , piece_id=JQ2C6EBHEGTYAZGA3D2XXWSWKM4SK6FKC7BQRVBQWUK3I6ZJODVA, seg_pos={0 0}, upload=356245]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: rpc: dial tcp 104.36.82.35:4615: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:07:48.027935077 (444h56m14.954229775s) [seg_pos={0 0}, upload=356245]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 11 22:07:48.053955429 (444h56m14.980250123s) [upload=356245]: Done uploading segments... Jul 11 22:07:48.053986707 (444h56m14.980281386s) [upload=356245]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 720.598002ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................X..XX.XXXXX......................] duration: 549.619516ms FAIL (2025-07-11 22:24:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 111885206s) [seg_pos={0 0}, upload=356247]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 11 22:07:23.210158850 (444h55m50.136453549s) [upload=356247]: Done uploading segments... Jul 11 22:07:23.210206783 (444h55m50.13650146s) [upload=356247]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 11 22:07:49.462605810 (444h56m16.388900487s) [upload=356257]: Starting upload Jul 11 22:07:49.462711957 (444h56m16.389006672s) [upload=356257]: Done starting upload Jul 11 22:07:49.463056633 (444h56m16.38935132s) [upload=356257]: Uploading segments... Jul 11 22:07:49.463098578 (444h56m16.38939327s) [upload=356257]: Got next segment. Inline: false Jul 11 22:07:49.463155286 (444h56m16.389449961s) [upload=356257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 11 22:07:49.463161111 (444h56m16.389455787s) [upload=356257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 11 22:07:49.491310937 (444h56m16.417605634s) [seg_pos={0 0}, upload=356257]: Begin upload segment... Jul 11 22:07:49.491376913 (444h56m16.417671608s) [seg_pos={0 0}, upload=356257]: Done begin upload segment. Jul 11 22:07:49.491418645 (444h56m16.417713343s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491424884 (444h56m16.417719569s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491891336 (444h56m16.41818603s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.491932150 (444h56m16.418226838s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492297977 (444h56m16.418592676s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492708109 (444h56m16.419002806s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.492831692 (444h56m16.419126389s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493140400 (444h56m16.419435094s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493513089 (444h56m16.419807785s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493859866 (444h56m16.420154562s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.493873602 (444h56m16.420168296s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494237372 (444h56m16.420532051s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494279553 (444h56m16.42057447s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494691432 (444h56m16.420986125s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.494952787 (444h56m16.421247476s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.495309973 (444h56m16.421604662s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Uploading piece... Jul 11 22:07:49.583944821 (444h56m16.510239514s) [upload=356257]: Next returned nil segment Jul 11 22:07:49.583963891 (444h56m16.51025859s) [upload=356257]: Waiting for error group managing segments... Jul 11 22:07:49.591301666 (444h56m16.517596353s) [noise=true , address=45.91.116.11:4117 , piece_id=7REXSDEG3TJR6C7K7LXRMRQIESHQPE3JG76S63K4ERUKZ4SZIGHA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592452807 (444h56m16.518747487s) [noise=true , address=45.91.116.7:4011 , piece_id=KS54OD347VYMEXUVDEE57IDDNWTFUDTK6MXGO2AHWYYPQ2D27T5Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.592568468 (444h56m16.518863145s) [noise=true , address=45.91.116.11:4119 , piece_id=2C2UBEGDJBV7QRCSMG42P22CPR7RSUAS2AFQA6CHJQUTGC3J7V3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.593965096 (444h56m16.520259769s) [noise=true , address=45.91.116.8:4048 , piece_id=4EFFLBDFV4HXDIDMDQ226JEJZYPVGWKHL7BJYR2FG5IVDCKDUM3Q, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.617606427 (444h56m16.543901117s) [noise=true , address=45.91.116.10:4082 , piece_id=FP7626NWXZNI3XGAT3WRMHKD4EK2TTT4F27Y4AC6OHJYHOZZSRWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:49.794074436 (444h56m16.720369123s) [noise=true , address=45.91.116.14:4191 , piece_id=D576HGUA6CO5EM6OVVXUDCO7RCT75VBMTDFYPGGJEQJIES75NNOA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.063137077 (444h56m16.989431755s) [noise=true , address=45.91.116.10:4083 , piece_id=6ILWQLY5RRKXKY4C37GK4CJPCX6H3MNKYPTWH5XHHRTJNR2ZA4EQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.127180651 (444h56m17.053475331s) [noise=true , address=77.90.31.15:4697 , piece_id=UOLPF2WE453V2Q2HXXNVVIPI77JEISTXZFCN4IKHMUHMTF3QZTVQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.381780303 (444h56m17.308074991s) [noise=true , address=77.90.31.18:4806 , piece_id=HNFFMRG4DITLJFQN4D3WV6YN7J2TOVWOWBFS2JQ2LWZIHGX3RREA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:07:50.392562323 (444h56m17.318857004s) [noise=true , address=77.90.31.21:4877 , piece_id=JAK4WYMMWYR2564PC62RGEEFKVSZXL4JTPPWEP6EJRYSDJH4BOMA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: <nil> Jul 11 22:08:19.432958186 (444h56m46.359252883s) [noise=true , address=104.36.82.4:4801 , piece_id=AR6ORHCLSUU7SPOURBRS7OR5JJE3UPWZ6EWFLYBI4IYKFOUMVCPA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433138461 (444h56m46.35943315s) [noise=true , address=104.36.82.16:4294 , piece_id=Q6XMNR3XFW42YFP5HJ3HA3P753HSECPTBU4UCIUQS25FADYFGYGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433278369 (444h56m46.359573062s) [noise=true , address=104.36.82.13:4186 , piece_id=TB7KMJNSFGOQQML24Z5MGDRSD5XCJJPCR5PMEX5GFWJXMKJPSEGA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433434524 (444h56m46.35972921s) [noise=true , address=104.36.82.6:4876 , piece_id=AJARLQARFLVUH4YJURFOG4SNKZMHKSEGAXANC5V4OZFAABTFNPJA, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 159.223.143.39:45512->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433519901 (444h56m46.359814588s) [noise=true , address=104.36.82.11:4114 , piece_id=QBNR5G3JI55PXOIHOPHNRASBYTI6HF4IIAAC35QDFJ2KJMYH3SCQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433559255 (444h56m46.359853933s) [noise=true , address=104.36.82.12:4151 , piece_id=OYKU2X3LG3PXLQASI6N3VTASA5CXIW2GACSFGDLTBMIKGIRR3ZWQ, seg_pos={0 0}, upload=356257]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 159.223.143.39:40624->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 11 22:08:19.433582918 (444h56m46.35987761s) [seg_pos={0 0}, upload=356257]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 11 22:08:19.462412108 (444h56m46.388706797s) [upload=356257]: Done uploading segments... Jul 11 22:08:19.462440667 (444h56m46.388735358s) [upload=356257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 549.648867ms core uplink us-select-1

1
...
2770
2771
2772
2773
2774
...
2999