Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X................X..........] duration:
27.594695126s
FAIL (2025-06-22 22:51:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.594731524s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
34.193290293s
FAIL (2025-06-22 22:50:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 34.193367822s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................X................X.........] duration:
34.366229037s
FAIL (2025-06-22 22:50:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 34.366267614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
16.854152733s
FAIL (2025-06-22 22:50:04 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 16.854186664s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................X................X........] duration:
18.992551589s
FAIL (2025-06-22 22:49:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.992579568s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
14.665499396s
FAIL (2025-06-22 22:49:37 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.665537343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................X................X.......] duration:
15.833235097s
FAIL (2025-06-22 22:49:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.833263811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
30.875396771s
FAIL (2025-06-22 22:48:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 30.875424499s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [....................................X................X......] duration:
37.453744408s
FAIL (2025-06-22 22:48:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 37.453847209s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
14.630801192s
FAIL (2025-06-22 22:47:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.630825525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X................X.....] duration:
42.598162346s
FAIL (2025-06-22 22:47:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 42.59819177s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
27.564075405s
FAIL (2025-06-22 22:47:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 27.564096642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.02390793s
FAIL (2025-06-22 22:46:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
310 (102h22m26.488716309s) [upload=81892]: Starting upload
Jun 22 22:45:50.925469211 (102h22m26.48879221s) [upload=81892]: Done starting upload
Jun 22 22:45:50.925583170 (102h22m26.48890617s) [upload=81892]: Uploading segments...
Jun 22 22:45:50.925731919 (102h22m26.489054923s) [upload=81892]: Got next segment. Inline: false
Jun 22 22:45:50.925812139 (102h22m26.489135124s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jun 22 22:45:50.925819319 (102h22m26.489142322s) [upload=81892]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jun 22 22:45:50.966720480 (102h22m26.530043504s) [seg_pos={0 0}, upload=81892]: Begin upload segment...
Jun 22 22:45:50.966790575 (102h22m26.530113567s) [seg_pos={0 0}, upload=81892]: Done begin upload segment.
Jun 22 22:45:50.966845202 (102h22m26.530168215s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.966908099 (102h22m26.530231128s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967478534 (102h22m26.530801545s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967543799 (102h22m26.530866805s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.967960433 (102h22m26.5312835s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968106903 (102h22m26.531429912s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968373261 (102h22m26.531696263s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968691298 (102h22m26.532014301s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.968773795 (102h22m26.532096799s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969347159 (102h22m26.532670165s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969416409 (102h22m26.532739418s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.969712281 (102h22m26.533035284s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970139738 (102h22m26.533462742s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970201415 (102h22m26.533524425s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.970708256 (102h22m26.534031266s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971216901 (102h22m26.534539912s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.971720308 (102h22m26.535043313s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:50.972263327 (102h22m26.535586335s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Uploading piece...
Jun 22 22:45:51.051223491 (102h22m26.614546495s) [upload=81892]: Next returned nil segment
Jun 22 22:45:51.051243508 (102h22m26.614566518s) [upload=81892]: Waiting for error group managing segments...
Jun 22 22:45:51.074288273 (102h22m26.637611277s) [noise=true , address=79.127.220.98:4410 , piece_id=Y2FDTZBLKMJO25WVOEYAY3R4JFXQOIGZ5YIUYVPJALVVFBXFOYKQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:1Lay6oZEhDEeDmmkisrLND1j82PurLHV95nGJoKkk5wKUaK5S7, address:79.127.220.98:4410): protocol: storage node overloaded, request limit: 500
Jun 22 22:45:51.103512317 (102h22m26.666835306s) [noise=true , address=104.36.82.21:6483 , piece_id=SU2PTJKR4DGNFIJN4ND3PQTYU6N3XQSY5NXNEDQIU3YN7HDT7CUA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12gxKuBALDLDVDfRGBvMZymwGyPfPtV1BsHdBPqZs34ugvvYacG, address:104.36.82.21:6483): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.318023918 (102h22m26.881346918s) [noise=true , address=104.36.82.53:4139 , piece_id=VPXFX5DYTL7ZXTKCGX4V4CILATFBIVI4A4UJRME35ZO5RZWNEPFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12T99BLYzWYXV5VkEKAYQVyzoeHA4XYz4Jrh8Dz3K6jqkec52Lg, address:104.36.82.53:4139): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:51.678434066 (102h22m27.241757059s) [noise=true , address=77.90.31.21:4888 , piece_id=RE5RNYL36XG5YG4U7TLRUJBAFV2UKUGDQGP2GDTGYP26HDDG4VJQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.207676865 (102h22m27.770999857s) [noise=true , address=77.90.31.14:4696 , piece_id=GBW6GN5DXS4Y4LKNAZWPYSI75THRXRHICL2CF2UMHW6HCJJPPTSQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.463664302 (102h22m28.026987384s) [noise=true , address=104.36.82.52:4111 , piece_id=UYPP46464DOGQWLRH67HCSPZRTIWRJ3CVYY6LN5GA7XTFIMW2DRA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:52.530920011 (102h22m28.09424301s) [noise=true , address=45.91.116.4:4965 , piece_id=SHMUF426X53VJGGKTAXTZTF56AUAKG6QP57YONZ4C4HEOMWAEM2A, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12cKuthv2wScEvcdmCxmPiQGGZ5w4tJ5a6tbJdBKt1SJ4e9Cszi, address:45.91.116.4:4965): protocol: storage node overloaded, request limit: 2000
Jun 22 22:45:52.796897000 (102h22m28.360220001s) [noise=true , address=77.90.31.19:4856 , piece_id=LMNSI6ZZ6FSNH2ILUVZTQB4N34D2GQEVU4HN2NXZGEYGVRR6UUVA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:55.633666704 (102h22m31.196989696s) [noise=true , address=77.90.31.15:4706 , piece_id=4K2G76FFG445VX33OQDTAHK37GVSGTAMSEAMRYAST5RYAFRQVSYA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:45:58.265206332 (102h22m33.828529324s) [noise=true , address=77.90.31.12:4620 , piece_id=RDQ4X5B6GAOFPMR7KTCQFZIR35XMIDUK3UZFHW5CKURZBDG647YA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:01.921320896 (102h22m37.484643884s) [noise=true , address=104.36.82.51:4076 , piece_id=E6IMG2SXQWIMUNBUJUFSBYHZYQO3B2PHAD564P2GGNL226OJNSMQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:02.038042402 (102h22m37.601365396s) [noise=true , address=104.36.82.13:6209 , piece_id=W47GRDELSPPRVHVU47EOTLMWZHLTGJMS2YYYCIPTUTOIEWM77OMA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:13.682107076 (102h22m49.245430077s) [noise=true , address=104.36.82.54:4189 , piece_id=36XN6O6NUCMWOJVIADZPIRODYIHOXZ3XXQRDTITV57ON7L5ZUA2Q, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:14.073097229 (102h22m49.636420251s) [noise=true , address=104.36.82.12:6158 , piece_id=VIVWFGI57SWJV5KFZZI345NKUMGZYNC2EYNIRX3I6CDWIJ6OISPA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:18.418008993 (102h22m53.981331998s) [noise=true , address=104.36.82.6:5882 , piece_id=DPQBXKZ2MMQ2MGXNRF74AFV7RZJCF2ZSAGKJGHYB5LIRK4HCHCJA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:20.140586391 (102h22m55.7039094s) [noise=true , address=104.36.82.11:6119 , piece_id=JD5ANZZUVU5XMBLHSODPUVF73QD2X2AKHTMDGP27XIP6XEQJUNEA, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:24.486429456 (102h23m0.049752457s) [noise=true , address=104.36.82.56:4247 , piece_id=NLYOEOAAIRDSFDUVI7KV2C5IYS4NDPBNKT7JW2GIRTK6I7U6FWFQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: <nil>
Jun 22 22:46:50.891509230 (102h23m26.454832228s) [noise=true , address=104.36.82.55:4215 , piece_id=W4YWI36H5I463IF5OQU5DWL4H6JD4BBSCKU2JQ7V75QMRHPT6KWQ, seg_pos={0 0}, upload=81892]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; write tcp 159.223.143.39:40930->104.36.82.55:4215: use of closed network connection; piecestore: piecestore close: context deadline exceeded
Jun 22 22:46:50.891544193 (102h23m26.454867196s) [seg_pos={0 0}, upload=81892]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Jun 22 22:46:50.913428886 (102h23m26.476751898s) [upload=81892]: Done uploading segments...
Jun 22 22:46:50.913455303 (102h23m26.476778302s) [upload=81892]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.023924054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [......................................X................X....] duration:
28.049737284s
FAIL (2025-06-22 22:46:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.04977082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X................X...] duration:
19.557806092s
FAIL (2025-06-22 22:45:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.558670387s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
819.529561ms
FAIL (2025-06-22 22:45:04 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 819.560052ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [........................................X................X..] duration:
25.059511477s
FAIL (2025-06-22 22:44:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.059532671s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
803.544658ms
FAIL (2025-06-22 22:44:07 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 803.576784ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X................X.] duration:
26.584225206s
FAIL (2025-06-22 22:43:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.584266216s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X................X] duration:
1m0.001480583s
FAIL (2025-06-22 22:43:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001501295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
840.578297ms
FAIL (2025-06-22 22:42:59 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 840.608023ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
794.515333ms
FAIL (2025-06-22 22:42:37 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 794.611393ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
15.698568455s
FAIL (2025-06-22 22:41:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.698584861s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
800.060313ms
FAIL (2025-06-22 22:41:23 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 800.085359ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
57.027994971s
FAIL (2025-06-22 22:41:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 57.028019536s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
26.173965698s
FAIL (2025-06-22 22:40:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.173992561s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
827.578284ms
FAIL (2025-06-22 22:40:16 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 827.612386ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
26.156675961s
FAIL (2025-06-22 22:39:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 26.1567012s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
836.247046ms
FAIL (2025-06-22 22:39:06 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 836.273112ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
20.720848155s
FAIL (2025-06-22 22:38:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.720862194s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
919.627774ms
FAIL (2025-06-22 22:38:19 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 919.656452ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
20.957903247s
FAIL (2025-06-22 22:38:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.957943692s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
860.905226ms
FAIL (2025-06-22 22:37:52 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 860.957975ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
835.494102ms
FAIL (2025-06-22 22:37:31 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 835.520834ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
28.307990969s
FAIL (2025-06-22 22:37:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 28.308012922s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
802.760272ms
FAIL (2025-06-22 22:36:57 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 802.793825ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
20.924271848s
FAIL (2025-06-22 22:36:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.924295617s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
859.583117ms
FAIL (2025-06-22 22:36:08 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 859.609404ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
15.625925927s
FAIL (2025-06-22 22:35:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.625956369s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
832.436711ms
FAIL (2025-06-22 22:35:20 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 832.464461ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
933.664449ms
FAIL (2025-06-22 22:35:00 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 933.699803ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
22.114582199s
FAIL (2025-06-22 22:34:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.114609856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
841.746455ms
FAIL (2025-06-22 22:34:23 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 841.82215ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
13.330896111s
FAIL (2025-06-22 22:33:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.330925611s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
901.877763ms
FAIL (2025-06-22 22:33:57 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 901.90219ms
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
845.857579ms
FAIL (2025-06-22 22:33:39 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 845.888629ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
20.314922828s
FAIL (2025-06-22 22:33:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.314952747s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
828.736978ms
FAIL (2025-06-22 22:32:52 UTC)
orbiter warning: uplink: bucket: metaclient: context deadline exceeded
Duration: 828.77242ms
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
29.872538135s
FAIL (2025-06-22 22:32:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.872561069s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
18.318037868s
FAIL (2025-06-22 22:31:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.318074991s
core
uplink
us-select-1
us1-placement-20
|