Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 13.476444506s FAIL (2025-07-09 21:08:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.476476905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 34.281761142s FAIL (2025-07-09 21:08:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.28184433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......................................XX...............] duration: 493.234184ms FAIL (2025-07-09 21:08:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 493.301362ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 36.021079366s FAIL (2025-07-09 21:08:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.021101346s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 30.645518217s FAIL (2025-07-09 21:08:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 30.6455684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......................................XX..............] duration: 523.482805ms FAIL (2025-07-09 21:07:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 523.520861ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 30.823662404s FAIL (2025-07-09 21:07:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.823683632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 25.079163018s FAIL (2025-07-09 21:07:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.079187021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......................................XX.............] duration: 1.56310153s FAIL (2025-07-09 21:07:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.563119491s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 24.655984803s FAIL (2025-07-09 21:06:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.656008484s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 22.309247658s FAIL (2025-07-09 21:06:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.309294281s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.....X.......................................XX............] duration: 510.278547ms FAIL (2025-07-09 21:06:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 510.307672ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 19.818700428s FAIL (2025-07-09 21:05:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.819763102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 26.826157361s FAIL (2025-07-09 21:05:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.826188056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 39.794262879s FAIL (2025-07-09 21:05:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.794291268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.....X.......................................XX...........] duration: 535.574697ms FAIL (2025-07-09 21:04:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 535.603135ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 41.010697749s FAIL (2025-07-09 21:04:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 41.01073184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.....X.......................................XX..........] duration: 492.919181ms FAIL (2025-07-09 21:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 492.942935ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 34.335836705s FAIL (2025-07-09 21:04:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.33639799s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 15.818011428s FAIL (2025-07-09 21:04:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.818039511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.....X.......................................XX.........] duration: 13.398748643s FAIL (2025-07-09 21:04:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.398784641s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 27.423567192s FAIL (2025-07-09 21:03:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.423598021s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 31.440542868s FAIL (2025-07-09 21:03:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.440584979s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.....X.......................................XX........] duration: 16.739194517s FAIL (2025-07-09 21:03:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.739217849s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 14.798178736s FAIL (2025-07-09 21:02:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.798212535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 23.717138358s FAIL (2025-07-09 21:02:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.717187008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.....X.......................................XX.......] duration: 6.446543254s FAIL (2025-07-09 21:02:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.44657135s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 17.820943159s FAIL (2025-07-09 21:02:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.820972175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...................................................X.......] duration: 26.910499958s FAIL (2025-07-09 21:01:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.910518727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.....X.......................................XX......] duration: 8.317216212s FAIL (2025-07-09 21:01:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.317269417s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 28.23135795s FAIL (2025-07-09 21:01:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.231378673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...................................................X......] duration: 34.62881312s FAIL (2025-07-09 21:01:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.628848293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 26.174255151s FAIL (2025-07-09 21:00:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.174291437s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.....X.......................................XX.....] duration: 3.772667355s FAIL (2025-07-09 21:00:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.772695878s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...................................................X.....] duration: 31.464114432s FAIL (2025-07-09 21:00:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.464161322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.....X.......................................XX....] duration: 15.224473375s FAIL (2025-07-09 21:00:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.22449768s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...................................................X....] duration: 38.546057741s FAIL (2025-07-09 21:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.546082902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 20.741718358s FAIL (2025-07-09 20:59:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.741748787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.....X.......................................XX...] duration: 499.539136ms FAIL (2025-07-09 20:59:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.5885ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................................X...] duration: 31.076318739s FAIL (2025-07-09 20:59:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.076346684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.....X.......................................XX..] duration: 2.080562584s FAIL (2025-07-09 20:58:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.080585616s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 16.946990232s FAIL (2025-07-09 20:58:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.94703592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................................X..] duration: 30.436113599s FAIL (2025-07-09 20:58:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.436142483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.....X.......................................XX.] duration: 582.254738ms FAIL (2025-07-09 20:58:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.284195ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 34.854260585s FAIL (2025-07-09 20:57:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.854297308s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................................X.] duration: 37.75494997s FAIL (2025-07-09 20:57:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 37.754972213s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.....X.......................................XX] duration: 30.001194537s FAIL (2025-07-09 20:57:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001218572s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 1m0.001616056s FAIL (2025-07-09 20:56:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001646879s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 17.862303011s FAIL (2025-07-09 20:56:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.862334093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....X.......................................X] duration: 30.001412941s FAIL (2025-07-09 20:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001443231s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)