Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.............X..........] duration:
450.352175ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:19:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 450.380058ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
591.229328ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:19:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 591.253029ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................X....X.................................] duration:
536.871498ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:19:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 536.902166ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
618.714793ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:18:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 618.739106ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X.............X.........] duration:
494.434314ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:18:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 494.459342ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................X...X.............X............X.....] duration:
501.491622ms trace:
https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6
FAIL (2025-10-31 21:18:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.516035ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
481.780623ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:18:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 481.876877ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
571.670867ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:18:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 571.697707ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X.............X........] duration:
529.311216ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:18:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 529.339991ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................X....X................................] duration:
547.823161ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:18:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 547.846327ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
462.325888ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:18:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 462.365796ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X.............X.......] duration:
520.922173ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:17:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 520.969895ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................X...X.............X............X....] duration:
452.30396ms trace:
https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6
FAIL (2025-10-31 21:17:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 452.330221ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
430.078255ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:17:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 430.107108ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
585.537375ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:17:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 585.559857ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................X....X...............................] duration:
484.473081ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:17:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 484.524036ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
491.750946ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:17:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 491.777906ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................X....X..............................] duration:
502.824319ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:17:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.858714ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X.............X......] duration:
388.836001ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:17:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 388.865577ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................X...X.............X............X...] duration:
463.215754ms trace:
https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6
FAIL (2025-10-31 21:17:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 463.249979ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
756.964373ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:16:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 757.110569ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
579.81642ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:16:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 579.838938ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................X....X.............................] duration:
519.262194ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:16:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 519.281034ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X.............X.....] duration:
441.078112ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:16:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 441.106922ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
419.242664ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:15:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 419.269728ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................X...X.............X............X..] duration:
476.919527ms trace:
https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6
FAIL (2025-10-31 21:15:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 477.013304ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
886.521081ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:15:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 886.624746ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X.............X....] duration:
501.127348ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:15:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 501.151896ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
865.144029ms trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:15:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 865.169671ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
453.824623ms trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:15:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 453.854866ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................X....X............................] duration:
486.966036ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:15:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 486.993013ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X.............X...] duration:
518.780021ms trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:15:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 518.859332ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................X...X.............X............X.] duration:
499.249417ms trace:
https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6
FAIL (2025-10-31 21:15:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 499.27134ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.239322241s trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:14:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.239334906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001524828s trace:
https://jaeger.dev.storj.io/trace/2ac5649d33010a8f
FAIL (2025-10-31 21:14:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001547725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.082444566s trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:14:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1.082464527s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.028975207s trace:
https://jaeger.dev.storj.io/trace/31df3bc6bd52e22d
FAIL (2025-10-31 21:14:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
Oct 31 21:13:28.892758377 (33h33m59.556707582s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.892866006 (33h33m59.556814978s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893492689 (33h33m59.557441671s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.893827431 (33h33m59.557776423s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894092825 (33h33m59.558041794s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894282435 (33h33m59.558231418s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894591534 (33h33m59.558540522s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.894676793 (33h33m59.558625772s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895096400 (33h33m59.55904539s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895174003 (33h33m59.559122986s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895555889 (33h33m59.559504874s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.895943650 (33h33m59.559892629s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896342403 (33h33m59.560291387s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896796866 (33h33m59.560745851s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.896996650 (33h33m59.560945636s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897530698 (33h33m59.56147968s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.897970918 (33h33m59.5619199s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Uploading piece...
Oct 31 21:13:28.969377937 (33h33m59.633326911s) [upload=34889]: Next returned nil segment
Oct 31 21:13:28.969396291 (33h33m59.633345279s) [upload=34889]: Waiting for error group managing segments...
Oct 31 21:13:28.977452157 (33h33m59.641401137s) [noise=true , address=145.79.227.42:4136 , piece_id=5L56IVSHW3PCK5F74NT6TD2W7AEVZP3YJMOVCY2CMQZB2JSCLBNQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.981892143 (33h33m59.645841122s) [noise=true , address=145.79.227.39:4014 , piece_id=NE7HFPZT5NBIP3KIR4DHPKHTGU7LI64HN445JWRDU3RHEDP2IHGA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:28.989355135 (33h33m59.653304114s) [noise=true , address=145.79.227.33:4892 , piece_id=KXMFT7TDDHASA35OIXMKOE3EEONTISXTT7KNOMHJIZP6YI55EOWA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000219795 (33h33m59.664168809s) [noise=true , address=145.79.227.12:4798 , piece_id=QOTS2OL7SLUI6GCATSIRJ7ENOZWNPEWMPKILRI36AYFE6B2XDGCQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.000791996 (33h33m59.664740997s) [noise=true , address=145.79.227.35:4495 , piece_id=SWKETE525VJJ6RLXYAQWKCDPTUGV6Q53UM4K2XFZVC4QVDRZVLYQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.001410180 (33h33m59.66535916s) [noise=true , address=145.79.227.8:4353 , piece_id=VVOY3HVHZZJXQV4GFVHP44N43FDGGUVBQKEIL5RGIHYIGHJD6W4A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.007459084 (33h33m59.671408067s) [noise=true , address=145.79.227.49:4345 , piece_id=NE4ORNVELRHFULQVU6BU6C2R6WRYOZG2G5YDAABUAVEYJOUGOG5Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.008939693 (33h33m59.672888672s) [noise=true , address=145.79.227.38:4988 , piece_id=F2UCL2ST7JLDTKMAZJFZMZ6GXLY6M7D7NDRE7B6FBRGVDOTTI4OQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.011813843 (33h33m59.675762826s) [noise=true , address=145.79.227.14:4855 , piece_id=V3ZI2J256ZX3MYHHIPW6AG4O6VDBUOUKCHV6WPWQXRHKPXZRH3KQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.015740959 (33h33m59.679689942s) [noise=true , address=145.79.227.5:4234 , piece_id=X3YHSAZG5M3N6SHGTQ2FYHBKL4UPI6553HFT3SEILFWHAFBCSDMA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:13:29.322562407 (33h33m59.986511376s) [noise=true , address=145.79.227.36:4537 , piece_id=KYFWN6N4QQFSJNMT3C25I7EILT5NBSWJPLZ6UTBPGON2QQVRZI3Q, seg_pos={0 0}, upload=34889]: Done uploading piece. err: <nil>
Oct 31 21:14:28.833656225 (33h34m59.49760521s) [noise=true , address=77.90.31.12:4604 , piece_id=2KLVKCSL5MWEOUXTX22L6UC2JBWAXZRN4IS6J5SDN6TU2FGKUK3A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1vK9QCCj1rszVCR72YBc8BdKCPRno49RrcH9uGQb9rBJkZY4F7, address:77.90.31.12:4604): protocol: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:48482->77.90.31.12:4604: use of closed network connection
Oct 31 21:14:28.833865698 (33h34m59.497814683s) [noise=true , address=77.90.31.21:4896 , piece_id=PGQXCKIJUOPVWL4IX3YXGFFS7NNLRMEDPBT3S7Q25AV7S5KKEP7A, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:51998->77.90.31.21:4896: use of closed network connection
Oct 31 21:14:28.833883590 (33h34m59.497832575s) [noise=true , address=77.90.31.16:4747 , piece_id=W3KHIQKD567GA3EWBMY3BKSLNJ33GQK6K5Y3W6B77WULYPNMCVPQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, address:77.90.31.16:4747): protocol: read tcp 159.223.143.39:46506->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834135973 (33h34m59.498084955s) [noise=true , address=45.91.116.11:4133 , piece_id=46TV6O7JGJUXDWX2S5II4OMRNPQHTEYTDLYU2QC6K4R3MRX4BFUA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12e1wX7hiER8sX4MLm5qLue4mayiN4SxXFdZSxfymRaRZxRGU6E, address:45.91.116.11:4133): protocol: read tcp 159.223.143.39:32946->45.91.116.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834376742 (33h34m59.498325733s) [noise=true , address=77.90.31.18:4816 , piece_id=D4VNNMRQUBEB3U6THLLZFHCYZIZPH2XIZOPZG37USGYYY5FEYDYA, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: read tcp 159.223.143.39:35702->77.90.31.18:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834420496 (33h34m59.498369484s) [noise=true , address=77.90.31.17:4795 , piece_id=D33GMXNMRNT6725NTCXAQNXW5J6Y5CSOKKGNS5AKO5F332LBVNFQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: read tcp 159.223.143.39:53498->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:14:28.834810632 (33h34m59.498759611s) [noise=true , address=77.90.31.11:4562 , piece_id=YOQB5IVBUAB5MY6LSJJS66CQB4KOTCIYGNIKPPVPCX533VXYNQTQ, seg_pos={0 0}, upload=34889]: Done uploading piece. err: ecclient: upload failed (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r, address:77.90.31.11:4562): protocol: context deadline exceeded; context deadline exceeded
Oct 31 21:14:28.834844937 (33h34m59.498793907s) [seg_pos={0 0}, upload=34889]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Oct 31 21:14:28.860313861 (33h34m59.524262843s) [upload=34889]: Done uploading segments...
Oct 31 21:14:28.860352946 (33h34m59.524301935s) [upload=34889]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.029003955s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X.............X..] duration:
4.079660388s trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:14:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 4.079700502s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................X...X.............X............X] duration:
30.000693485s trace:
https://jaeger.dev.storj.io/trace/7e80ce77a39a74b6
FAIL (2025-10-31 21:14:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000712887s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X.............X.] duration:
1.757762515s trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:14:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 1.75778322s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X.............X] duration:
30.02609344s trace:
https://jaeger.dev.storj.io/trace/1f8ea7acb8ed6335
FAIL (2025-10-31 21:13:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
8]: Uploading segments...
Oct 31 21:13:28.316627400 (33h33m58.980576372s) [upload=34888]: Got next segment. Inline: false
Oct 31 21:13:28.316669888 (33h33m58.980618867s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:13:28.316690602 (33h33m58.980639584s) [upload=34888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:13:28.348216051 (33h33m59.01216503s) [seg_pos={0 0}, upload=34888]: Begin upload segment...
Oct 31 21:13:28.348261798 (33h33m59.012210778s) [seg_pos={0 0}, upload=34888]: Done begin upload segment.
Oct 31 21:13:28.348292760 (33h33m59.012241752s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348311865 (33h33m59.012260856s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.348901660 (33h33m59.012850648s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349417031 (33h33m59.013366014s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.349775696 (33h33m59.013724678s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350105744 (33h33m59.014054723s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350448959 (33h33m59.014397941s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.350811877 (33h33m59.014760863s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351198891 (33h33m59.015147872s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351564635 (33h33m59.015513615s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.351931566 (33h33m59.015880545s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352305495 (33h33m59.016254474s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.352701181 (33h33m59.016650163s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353155136 (33h33m59.01710412s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.353604338 (33h33m59.017553321s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.354118246 (33h33m59.018067236s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Uploading piece...
Oct 31 21:13:28.442410445 (33h33m59.106359434s) [upload=34888]: Next returned nil segment
Oct 31 21:13:28.442426080 (33h33m59.106375045s) [upload=34888]: Waiting for error group managing segments...
Oct 31 21:13:28.447614008 (33h33m59.111562999s) [noise=true , address=145.79.227.10:4403 , piece_id=BY4F7UYCBQFZ6U24STQBFNZOKO6FIUXEOKLQ2R5WLI72ZP7K7U7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.448391403 (33h33m59.112340386s) [noise=true , address=145.79.227.15:4873 , piece_id=5IF6JOBJKW65SVUPQCA3MU5BA2UJK46QZMI5P7OARO5N3OABQM4A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.451690172 (33h33m59.115639163s) [noise=true , address=145.79.227.11:4732 , piece_id=J7UHFMN7RTNSAJQIDM3WV2VBEB76TGAEHMXH2X6QQKKNBLSZGSAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452059710 (33h33m59.116008693s) [noise=true , address=145.79.227.31:4798 , piece_id=36QFKLNFEM4UZ2PFD6EFSZOCCGISVFLRUF4T5WBFR6447U4NO6WQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452148699 (33h33m59.11609768s) [noise=true , address=145.79.227.27:4690 , piece_id=E5ZMYBATGO2NKPPIBJFNAFFNPBW4HEBXQD44BVBUTCYG6AQHBVAQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452386518 (33h33m59.116335502s) [noise=true , address=145.79.227.17:4970 , piece_id=7TX5E6KUIAUP2YN2FVQMXSA7ESQC3YTTIJFF6AEFZFRLBGIXJYIQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.452791949 (33h33m59.116740931s) [noise=true , address=145.79.227.14:4839 , piece_id=RENB2RZRBWZWO7TM2QCBIGBJ6CGCJ6BBG5R2LQ57QSTHEUZM5CWA, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:28.486088942 (33h33m59.150037922s) [noise=true , address=145.79.227.18:4007 , piece_id=CXCPJKUAGM2QFS2ODAHS7EK2TZ3J557LO34REV4OVZKPJ7FPYLXQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: <nil>
Oct 31 21:13:58.284056384 (33h34m28.948005374s) [noise=true , address=45.91.116.9:4263 , piece_id=TDCDNBTU65Q7JJG4WJUPUAR2YDWU2VP4R4PDREDLU3GPGK5NISGQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:55678->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284204105 (33h34m28.948153087s) [noise=true , address=77.90.31.17:4771 , piece_id=FM2ERAHVUYIQEHR4GRCEQP4K2KIH22XC6ZFQR6UOOGR6T45WO2JQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:32938->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284272144 (33h34m28.948221135s) [noise=true , address=77.90.31.10:4517 , piece_id=XZE6R2Q3OWKXOBBUSUAZJ6GYAMK42DR2N3IFTSSG4V5WXQCLCLWQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: read tcp 159.223.143.39:60716->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284550548 (33h34m28.948499534s) [noise=true , address=77.90.31.18:4806 , piece_id=JQTIXCTDAAXUD3N3FLAQAQ3SVFUMIFIEX3URVU3IVD2LQJANOGTQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: read tcp 159.223.143.39:53164->77.90.31.18:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284751541 (33h34m28.948700528s) [noise=true , address=45.91.116.4:4939 , piece_id=MQQKFXGRBYWGN2FBOWADSIDUGHPRLZRKEY6O7JYNCKJKP6VZWLYQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:57178->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284907335 (33h34m28.948856315s) [noise=true , address=77.90.31.14:4662 , piece_id=2UW3DN5H5VIA2KVU6IDAG7Q2PHHBDV473RVYX4QZWSXQSF5I4F5A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:59098->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.284969486 (33h34m28.948918476s) [noise=true , address=77.90.31.12:4589 , piece_id=BCCD66TKU6IWIOCIO5JOJY3Y5ZAQDFUQIHI66X5KDTMZLMTYQAZQ, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 159.223.143.39:38302->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285124226 (33h34m28.949073215s) [noise=true , address=77.90.31.21:4878 , piece_id=UUXH5LIVGGS3NVQPJ4HUG5QHTPZD3Y3PKQT2RZE45HR6QPGRYC7A, seg_pos={0 0}, upload=34888]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:55670->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:13:58.285248033 (33h34m28.949197006s) [seg_pos={0 0}, upload=34888]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Oct 31 21:13:58.308728364 (33h34m28.972677346s) [upload=34888]: Done uploading segments...
Oct 31 21:13:58.308762041 (33h34m28.972711023s) [upload=34888]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.026114243s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................X....X...........................] duration:
1.960410456s trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:13:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.960430332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X...X.............X............] duration:
455.665542ms trace:
https://jaeger.dev.storj.io/trace/687e98b17351c5ae
FAIL (2025-10-31 21:12:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 455.68855ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
435.765423ms trace:
https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3
FAIL (2025-10-31 21:12:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload
Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload
Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments...
Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false
Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment...
Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment.
Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment
Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments...
Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments...
Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 435.794162ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................X....X..........................] duration:
593.471402ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:12:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 593.497834ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
492.34171ms trace:
https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3
FAIL (2025-10-31 21:11:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload
Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload
Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments...
Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false
Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment...
Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment.
Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment
Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments...
Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments...
Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 492.370409ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............................X....X.........................] duration:
482.455303ms trace:
https://jaeger.dev.storj.io/trace/58a21e4fe7f538ef
FAIL (2025-10-31 21:11:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 482.480227ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X...X.............X...........] duration:
492.870089ms trace:
https://jaeger.dev.storj.io/trace/687e98b17351c5ae
FAIL (2025-10-31 21:11:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 492.895864ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X...X.............X..........] duration:
460.395508ms trace:
https://jaeger.dev.storj.io/trace/687e98b17351c5ae
FAIL (2025-10-31 21:11:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.418112ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
460.328499ms trace:
https://jaeger.dev.storj.io/trace/72c96df3b6fa1df3
FAIL (2025-10-31 21:11:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Oct 31 21:03:19.259696991 (33h23m49.923645971s) [upload=34712]: Starting upload
Oct 31 21:03:19.259793353 (33h23m49.923742345s) [upload=34712]: Done starting upload
Oct 31 21:03:19.260026241 (33h23m49.923975274s) [upload=34712]: Uploading segments...
Oct 31 21:03:19.260800995 (33h23m49.924750077s) [upload=34712]: Got next segment. Inline: false
Oct 31 21:03:19.260904139 (33h23m49.924853128s) [upload=34712]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:03:19.260911785 (33h23m49.924860775s) [upload=34712]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:03:19.292259802 (33h23m49.95620879s) [seg_pos={0 0}, upload=34712]: Begin upload segment...
Oct 31 21:03:19.292329853 (33h23m49.956278835s) [seg_pos={0 0}, upload=34712]: Done begin upload segment.
Oct 31 21:03:19.292363333 (33h23m49.956312317s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.292382519 (33h23m49.956331511s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.293334418 (33h23m49.957283404s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.293931685 (33h23m49.95788067s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.294496338 (33h23m49.958445323s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.294954638 (33h23m49.958903608s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295156990 (33h23m49.959105977s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295555563 (33h23m49.959504553s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.295681255 (33h23m49.959630243s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296178767 (33h23m49.960127756s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296211168 (33h23m49.960160154s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296655953 (33h23m49.960604944s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.296749214 (33h23m49.960698184s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297289765 (33h23m49.961238741s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297308719 (33h23m49.961257708s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.297805757 (33h23m49.961754746s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Uploading piece...
Oct 31 21:03:19.386653410 (33h23m50.050602395s) [upload=34712]: Next returned nil segment
Oct 31 21:03:19.386670800 (33h23m50.050619792s) [upload=34712]: Waiting for error group managing segments...
Oct 31 21:03:19.393880229 (33h23m50.057829216s) [noise=true , address=145.79.227.20:4438 , piece_id=PL3ZXPXX3ZVVURXL6YPJSMTF3QO6OYCPXJSSOEO665TTK7BURL5Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394013373 (33h23m50.057962348s) [noise=true , address=145.79.227.21:4474 , piece_id=GMHXARWXIBYDGPRW5CAIZQZZVOJXZ5WP24JSTOGSQYTGKZRCNDXA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394577639 (33h23m50.058526624s) [noise=true , address=145.79.227.5:4223 , piece_id=TPC476NDULKFFB2JNS6CTQLEUF3QIWD2MI2SWDNPXONPZND6XWHQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.394734436 (33h23m50.058683415s) [noise=true , address=145.79.227.14:4837 , piece_id=C2DKOMYHIIX24YG6BR6I75IMEEEH62THTM4XIALZZJSL76OUITGA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395207093 (33h23m50.059156079s) [noise=true , address=145.79.227.4:4185 , piece_id=XXPUPBDCH5YQUCB4LINZCSIRCCSO3BCLDOO7NAIWSZLVXNV5PELA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395318900 (33h23m50.059267881s) [noise=true , address=145.79.227.9:4365 , piece_id=5IY5TR3BPMPFKTAOM6MCID75ELYLHFFOQWTDGPDI2F6PXWDZTGDQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.395692290 (33h23m50.059641274s) [noise=true , address=145.79.227.15:4874 , piece_id=OT2VS2CUZQC2JGKHF3TVJT4NZALIN3UXFDFUO4J6JXDM2AX5WQJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.396317319 (33h23m50.060266296s) [noise=true , address=145.79.227.10:4402 , piece_id=VHKDOUJS77FUHA7DD4K35ZHTR2JJOASZDGJXXQSXWCGACN7SFLOQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.396991407 (33h23m50.060940395s) [noise=true , address=145.79.227.32:4835 , piece_id=3IGQTZLFAWDKAGJCSFAQEPRIYUDB2GQPE3F4UDWRWFLWTIBSUBHA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.398213931 (33h23m50.06216292s) [noise=true , address=145.79.227.24:4583 , piece_id=4HKT753AAG3C6P3OA3EU6L4U4ODXQOIERZP4YYZHIW2G5OS25FJQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:19.403109883 (33h23m50.067058869s) [noise=true , address=145.79.227.16:4909 , piece_id=Y4AAYQ4RCTOSLJS4RUNIYUZ5Q7RN4X3CXDSLWDEQWDXCWVIF3YTQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: <nil>
Oct 31 21:03:49.224564837 (33h24m19.888513826s) [noise=true , address=77.90.31.14:4662 , piece_id=3VQY5YABGI6G2LJA2LPWDQDYV6TOVWPTSIJGPSD2GQTODQ5UVO3Q, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, address:77.90.31.14:4662): protocol: read tcp 159.223.143.39:51236->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.224761220 (33h24m19.888710206s) [noise=true , address=77.90.31.14:4663 , piece_id=IPCB4LKJYIOA2QJ3MNPRGZ74R445YEG3L5ZIZ7VQJJC7ZU4L74WA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: rpc: dial tcp 77.90.31.14:4663: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225000374 (33h24m19.888949424s) [noise=true , address=77.90.31.21:4878 , piece_id=DATWV64JQ36MISO33OW5676KDNQRZSMC7CUPQLP2TD67EVNOQPOA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 159.223.143.39:43804->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225245181 (33h24m19.88919417s) [noise=true , address=77.90.31.10:4519 , piece_id=LO6SX22EZQ34JX32SML7SOULR2EAKDXP6KZWOT57BD62ZMTWQDFA, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 159.223.143.39:47124->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225418032 (33h24m19.889367018s) [noise=true , address=45.91.116.9:4263 , piece_id=66EHLAWUTBIP5O3V6CHAFLHEE2D25JIZI5RRUG725RAFEIRDMOZQ, seg_pos={0 0}, upload=34712]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 159.223.143.39:48636->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:03:49.225451887 (33h24m19.889400867s) [seg_pos={0 0}, upload=34712]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Oct 31 21:03:49.248723404 (33h24m19.912672372s) [upload=34712]: Done uploading segments...
Oct 31 21:03:49.248871820 (33h24m19.912820789s) [upload=34712]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 460.359845ms
core
uplink
us-select-1
|