Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 35.47608507s FAIL (2025-07-09 21:36:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.476154514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 19.293429906s FAIL (2025-07-09 21:36:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.293488255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 18.809444154s FAIL (2025-07-09 21:36:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.809472045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 739.676078ms FAIL (2025-07-09 21:35:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 740.162189ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 36.597205684s FAIL (2025-07-09 21:35:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.597229197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 28.463919653s FAIL (2025-07-09 21:35:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 28.463959033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 892.365811ms FAIL (2025-07-09 21:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 892.390318ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 472.563815ms FAIL (2025-07-09 21:34:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 472.60488ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 36.325058776s FAIL (2025-07-09 21:34:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.325082867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 14.278538332s FAIL (2025-07-09 21:33:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.278569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 530.79622ms FAIL (2025-07-09 21:33:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.85563ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 29.593817233s FAIL (2025-07-09 21:33:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.593844434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 13.879691754s FAIL (2025-07-09 21:32:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.879722851s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 26.28842776s FAIL (2025-07-09 21:32:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 26.288453068s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 560.259969ms FAIL (2025-07-09 21:32:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 560.294946ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 14.120529245s FAIL (2025-07-09 21:31:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.120562226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 29.085648069s FAIL (2025-07-09 21:31:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.085675628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 721.381006ms FAIL (2025-07-09 21:31:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 721.409568ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 25.025039621s FAIL (2025-07-09 21:31:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.025061014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 12.850978975s FAIL (2025-07-09 21:30:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 12.851062017s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 19.784650382s FAIL (2025-07-09 21:30:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.784706134s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 643.405293ms FAIL (2025-07-09 21:30:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 643.429733ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 22.192537644s FAIL (2025-07-09 21:30:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.192591634s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 38.560378055s FAIL (2025-07-09 21:29:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.560409084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 17.594582652s FAIL (2025-07-09 21:29:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.594609485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 530.505467ms FAIL (2025-07-09 21:29:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.535416ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 14.367906679s FAIL (2025-07-09 21:29:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.367978627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 30.043441835s FAIL (2025-07-09 21:28:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.043460623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 51.091087932s FAIL (2025-07-09 21:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.091111611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 26.666655794s FAIL (2025-07-09 21:28:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.666688692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 1.118734428s FAIL (2025-07-09 21:28:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.118768518s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 38.311005751s FAIL (2025-07-09 21:27:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.311027081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 31.444000222s FAIL (2025-07-09 21:27:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 31.444046475s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 497.358093ms FAIL (2025-07-09 21:27:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 497.378531ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 486.84644ms FAIL (2025-07-09 21:26:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.891646ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 32.5770944s FAIL (2025-07-09 21:26:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.577190113s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 728.137514ms FAIL (2025-07-09 21:26:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 728.162821ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 13.896539429s FAIL (2025-07-09 21:25:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.896583508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 11.129853709s FAIL (2025-07-09 21:25:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.129886547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 32.565128005s FAIL (2025-07-09 21:25:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.56515432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 727.336544ms FAIL (2025-07-09 21:25:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 727.364719ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 39.342671192s FAIL (2025-07-09 21:24:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 39.342701056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 25.26596544s FAIL (2025-07-09 21:24:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.266313042s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 650.936532ms FAIL (2025-07-09 21:24:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 650.97289ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 29.072854399s FAIL (2025-07-09 21:24:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 29.072885031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 22.470238277s FAIL (2025-07-09 21:23:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.470253218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 717.824287ms FAIL (2025-07-09 21:23:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 717.859372ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 24.465998065s FAIL (2025-07-09 21:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.466020756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 23.932324126s FAIL (2025-07-09 21:22:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.932389583s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 573.457831ms FAIL (2025-07-09 21:22:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.487327ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)