Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 15.666477886s FAIL (2025-07-09 21:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.666585315s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 724.791796ms FAIL (2025-07-09 21:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 724.816911ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 17.020818477s FAIL (2025-07-09 21:22:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.020830931s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 16.437076522s FAIL (2025-07-09 21:22:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 16.437135273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 56.420661718s FAIL (2025-07-09 21:22:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.420685215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 16.414118643s FAIL (2025-07-09 21:21:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.414144086s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 783.6609ms FAIL (2025-07-09 21:21:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 783.699208ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 15.961603063s FAIL (2025-07-09 21:20:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.96163352s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 5.118632029s FAIL (2025-07-09 21:20:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.118654132s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 31.302959134s FAIL (2025-07-09 21:20:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 31.303105886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 17.304864682s FAIL (2025-07-09 21:20:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 17.305062981s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 906.851087ms FAIL (2025-07-09 21:19:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 906.875823ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 42.134340718s FAIL (2025-07-09 21:19:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.134361214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.984948331s FAIL (2025-07-09 21:19:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.984972036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 26.532636442s FAIL (2025-07-09 21:19:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.532660418s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 56.247560884s FAIL (2025-07-09 21:19:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 56.247615419s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 12.001813857s FAIL (2025-07-09 21:19:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.001850016s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 22.32752732s FAIL (2025-07-09 21:18:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.327555386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 22.720660632s FAIL (2025-07-09 21:17:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.720683746s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 22.41577619s FAIL (2025-07-09 21:17:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 22.415844317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 520.006283ms FAIL (2025-07-09 21:17:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.970831ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 42.378438819s FAIL (2025-07-09 21:17:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 42.378463266s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 33.571616783s FAIL (2025-07-09 21:17:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.571642759s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 21.005232418s FAIL (2025-07-09 21:16:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.00525538s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 25.74040307s FAIL (2025-07-09 21:16:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.740431669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 22.715381185s FAIL (2025-07-09 21:16:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 22.715412222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 9.982904539s FAIL (2025-07-09 21:15:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.982980753s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 571.912118ms FAIL (2025-07-09 21:15:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 571.938485ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 20.433354352s FAIL (2025-07-09 21:15:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.433395956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 44.073638256s FAIL (2025-07-09 21:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.073702268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 1.515749535s FAIL (2025-07-09 21:15:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.515780364s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 539.524187ms FAIL (2025-07-09 21:14:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 539.542804ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 11.853778774s FAIL (2025-07-09 21:14:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.853803092s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 35.059178181s FAIL (2025-07-09 21:14:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.059189969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 33.731353164s FAIL (2025-07-09 21:14:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 33.731388381s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 46.572929789s FAIL (2025-07-09 21:13:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 46.572956939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 1.812213066s FAIL (2025-07-09 21:13:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.812237566s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 710.921047ms FAIL (2025-07-09 21:12:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 710.939047ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 34.267928625s FAIL (2025-07-09 21:12:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 34.267960593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 20.643133261s FAIL (2025-07-09 21:12:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.643163192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 19.679984459s FAIL (2025-07-09 21:12:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 19.680011818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......................................XX..................] duration: 588.379952ms FAIL (2025-07-09 21:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 588.403358ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 24.463577627s FAIL (2025-07-09 21:11:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.463608362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 20.17543178s FAIL (2025-07-09 21:11:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.175453426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 42.437699476s FAIL (2025-07-09 21:11:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 42.437791774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 32.438206981s FAIL (2025-07-09 21:10:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.440745576s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......................................XX.................] duration: 564.764507ms FAIL (2025-07-09 21:10:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 564.790523ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................................XX................] duration: 504.137865ms FAIL (2025-07-09 21:09:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.166657ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.127715838s FAIL (2025-07-09 21:09:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload=316819]: Done starting upload Jul 9 20:51:01.744640664 (395h39m28.670935357s) [upload=316819]: Uploading segments... Jul 9 20:51:01.745376443 (395h39m28.671671133s) [upload=316819]: Got next segment. Inline: false Jul 9 20:51:01.745525760 (395h39m28.67182045s) [upload=316819]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 20:51:01.745533412 (395h39m28.671828085s) [upload=316819]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 20:51:01.783911087 (395h39m28.710205769s) [seg_pos={0 0}, upload=316819]: Begin upload segment... Jul 9 20:51:01.783977239 (395h39m28.710271934s) [seg_pos={0 0}, upload=316819]: Done begin upload segment. Jul 9 20:51:01.784009016 (395h39m28.710303716s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784481955 (395h39m28.710776649s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.784908723 (395h39m28.711203414s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785241082 (395h39m28.711535772s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785533991 (395h39m28.711828684s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.785835130 (395h39m28.71212983s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786163209 (395h39m28.712457905s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.786710678 (395h39m28.713005376s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.787413905 (395h39m28.713708604s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788027996 (395h39m28.714322693s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.788820165 (395h39m28.715114861s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789282608 (395h39m28.715577305s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.789720674 (395h39m28.716015375s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790132999 (395h39m28.716427699s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.790759994 (395h39m28.71705469s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795058621 (395h39m28.721353314s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.795650095 (395h39m28.721944823s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.798606176 (395h39m28.724900871s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Uploading piece... Jul 9 20:51:01.909993379 (395h39m28.836288058s) [upload=316819]: Next returned nil segment Jul 9 20:51:01.910016191 (395h39m28.83631089s) [upload=316819]: Waiting for error group managing segments... Jul 9 20:51:01.919116414 (395h39m28.845411096s) [noise=true , address=77.90.31.14:4678 , piece_id=HBT23VUKCRDCJWMP5UYHAJ52UPO23PP44VXAPXR7Q73ZMORPGPPQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.919780312 (395h39m28.846074986s) [noise=true , address=77.90.31.11:4578 , piece_id=CTRGOFGYL5GM5LHG34IRIDUEOY6RKZ3HZORKJW7XKVRBW55ZRE3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.920354582 (395h39m28.846649276s) [noise=true , address=77.90.31.21:4880 , piece_id=AGTABQXZT3L2NRSSXBZT7P7OVRA5TTSK4PPXITDFHQ46BQBODGHA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.921019660 (395h39m28.847314356s) [noise=true , address=77.90.31.17:4784 , piece_id=6NMQWGOJPBS5U76BMHCTI3RBXDHX3727ULH4US35W7IVXHAQPPQA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.922603097 (395h39m28.848897786s) [noise=true , address=77.90.31.13:4632 , piece_id=4PFLEHBC46MBNEWKJYKCZ66JXTQA7H5OTRHGGEMZQ33NNAVWFUMA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.932689000 (395h39m28.858983677s) [noise=true , address=77.90.31.19:4860 , piece_id=OM4TEL3C5R3N4KBHP5KRO6LBIB3LLXBCEKOEPRCSNKFT2H24FZBA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.934674660 (395h39m28.86096935s) [noise=true , address=77.90.31.15:4709 , piece_id=XFWO5EKOOSN6OBFIWI24F7TCRUSPMZIIN3DOEL2ERZ3VOITY773A, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:01.936194700 (395h39m28.862489378s) [noise=true , address=77.90.31.10:4523 , piece_id=SPWYFTHRZXVJVDII6CS2RUVLSVZOAGWKUJNCKU2VPSYRPTPEVO3Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:02.795832904 (395h39m29.722127608s) [noise=true , address=45.91.95.2:4009 , piece_id=WJMTOBG6FMEPZRVITBAYY2YSUEAQLYRG2VOVUVLDXJPXZ4GQQLZQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:1SrEo4XQ8dBq87CWMuQvExnt7TuL9JbwXqFSehpJ6pPsCHBAi6, address:45.91.95.2:4009): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:03.997629008 (395h39m30.923923684s) [noise=true , address=45.91.95.3:4033 , piece_id=GTYIDWLQNZDKKDTJ4WR23XUATGH2KJEO6AZOV6INJOPRNGZZY6XQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12p7yU1B5SZNrV43Fd8jJr2FSykKDV1QsVgTCNgcBMtB46DazUD, address:45.91.95.3:4033): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:04.680596617 (395h39m31.606891297s) [noise=true , address=45.91.116.5:4315 , piece_id=4DU4MG7F55AV65OBOLWNZW5WPRHWPT5JDLYCS3W7P22ITVQ4ONKA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:06.998485262 (395h39m33.924779955s) [noise=true , address=45.91.95.5:4012 , piece_id=K7V4QEUEO4QNA5ALS6C5R5ST5P5NXRP33MFN4X4JIBOVK7XUQZJA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:121sqZ4w3zPsKS2g6ufEhdqzBnZcws9558iAfpfrKxdEm6LhA5S, address:45.91.95.5:4012): protocol: storage node overloaded, request limit: 2000 Jul 9 20:51:22.922410916 (395h39m49.848705597s) [noise=true , address=104.36.82.23:4464 , piece_id=REM6P63TU6YH4FQNTUIMKSL34LFEKWJR2PNY364277A23GB47DYA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:29.420748222 (395h39m56.347042898s) [noise=true , address=45.91.95.4:4001 , piece_id=TMC2S53RJT4CZVVDNTXXKXOEVATBGSHCSUBU2NZLF6QWSBA7XZAQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:31.417155305 (395h39m58.343450001s) [noise=true , address=104.36.82.6:4906 , piece_id=JWKITIOBS27IKAOVSHXT4IM5EYX47UTAGHL6XANGDNK56DCPLHDA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:51:42.331922832 (395h40m9.258217513s) [noise=true , address=104.36.82.3:4782 , piece_id=SRHJ3DNPI4ZK6NRDMCNLYJR6MIHTGVMA4WZEVAP4FSSBWUVQKSEQ, seg_pos={0 0}, upload=316819]: Done uploading piece. err: <nil> Jul 9 20:52:01.708814311 (395h40m28.635109007s) [noise=true , address=104.36.82.19:4403 , piece_id=A2DCQOGRLU4SKPAFLL7RR7HRSSRH3TTPU7XIOWHO6AWOT3OWHXXA, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:104.36.82.19:4403): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 20:52:01.708928322 (395h40m28.635223001s) [noise=true , address=104.36.82.44:4892 , piece_id=NARCZA3HD2V5W7UZGLDAOBZ62ZQS243R2CSXG3XGDS4JMTRREN2Q, seg_pos={0 0}, upload=316819]: Done uploading piece. err: ecclient: upload failed (node:12RHwFxcUjPsn12uHEAH3fBDJKv1v1tM8mRJXoFrZrG5djdswF5, address:104.36.82.44:4892): protocol: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:57578->104.36.82.44:4892: use of closed network connection Jul 9 20:52:01.708993102 (395h40m28.635287793s) [seg_pos={0 0}, upload=316819]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 20:52:01.734227352 (395h40m28.66052204s) [upload=316819]: Done uploading segments... Jul 9 20:52:01.734255282 (395h40m28.660549975s) [upload=316819]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.127747577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 21.434735633s FAIL (2025-07-09 21:09:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.434830676s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)