Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........................XXX.X..............................] duration:
465.057769ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:49:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 465.074885ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................X....X...X........X...X...............] duration:
476.19771ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:48:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 476.225745ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................XXX.................................] duration:
524.324678ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:48:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 524.347011ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [...................................XXXXXXX.XXXXXXXXXXX.XXXXX] duration:
1m0.025136794s trace:
https://jaeger.dev.storj.io/trace/63596f586fb24a63
FAIL (2025-12-10 23:48:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
3438]: Uploading segments...
Dec 10 23:47:43.694837408 (128h19m44.522545813s) [upload=133438]: Got next segment. Inline: false
Dec 10 23:47:43.694890420 (128h19m44.522598821s) [upload=133438]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:47:43.694897354 (128h19m44.522605757s) [upload=133438]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:47:43.723466618 (128h19m44.551175003s) [seg_pos={0 0}, upload=133438]: Begin upload segment...
Dec 10 23:47:43.723532067 (128h19m44.551240452s) [seg_pos={0 0}, upload=133438]: Done begin upload segment.
Dec 10 23:47:43.723574836 (128h19m44.55128323s) [noise=true , address=145.79.227.2:4126 , piece_id=5NWC43ZT64OTLF7I3OJTAE5C6ZKWSKPQMLZY3FIB4ADNIEGFEJ7A, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.723630695 (128h19m44.551339097s) [noise=true , address=77.90.31.17:4787 , piece_id=B47TMDN4VCVUPUJHI7R7GBEHY3DPIXDV4RMN33ULB7S5R66K5YFA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.724159276 (128h19m44.551867669s) [noise=true , address=45.91.116.13:4181 , piece_id=PES4XRWCRUAAGUCW32A4ZW3ZRJ6JQWPU4G6E7ZXS6XQFWWEI7HSQ, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.724329681 (128h19m44.552038084s) [noise=true , address=45.91.116.10:4104 , piece_id=UTYNQK3VJTGSB6RB5GFDWCL4FCID6KKONMN7ABTVAY6LWJ6HIXMA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.724823235 (128h19m44.552531632s) [noise=true , address=145.79.227.33:4893 , piece_id=2GTGRATIIXJ53SFI3CKRSMEMQOYEGPKA4ODXQZE6XDUXIMMGSW2A, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.725170603 (128h19m44.552878997s) [noise=true , address=77.90.31.13:4660 , piece_id=IUFYQGDXWVX5ST77OZ3SPKYXOD72CYW5IX42TBXQOLX3DTAI7J2A, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.725535164 (128h19m44.553243556s) [noise=true , address=5.231.73.4:4028 , piece_id=FMUEVCR4LJFLC3QO6C4DWERVZUWR3MX4NQ3BUFSFQRUWWALPKNWA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.725990335 (128h19m44.553698739s) [noise=true , address=45.91.116.14:4195 , piece_id=5GDMRZS2YNFQTGQXVE2XG3EEJ2PORUXQATCNWMSJE26JYHRBHA7A, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.726418533 (128h19m44.554126936s) [noise=true , address=45.91.116.7:4025 , piece_id=Y4WVBL3ZMWLIB4U5OEY4WZZVS3THRFE7THC5ZMO73TZK7PGA6NCA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.726850860 (128h19m44.55455926s) [noise=true , address=77.90.31.18:4822 , piece_id=VKGXIPMO53OFOHZZDR6AZIW7UOSQMOXO3DCWU63IQTJZTMMRRMRQ, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.727338091 (128h19m44.55504649s) [noise=true , address=145.79.227.16:4935 , piece_id=UI2JPSHBXXKBSRLQ3LCZH2K6JX54N5XBYJA7HCMCTL27URJHKIYQ, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.727498623 (128h19m44.555207025s) [noise=true , address=77.90.31.14:4683 , piece_id=TL5IIMRX7SLBH77ZD5OLVIZ57VG455PK426NATNUBXOL746JBMMQ, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.727867026 (128h19m44.555575418s) [noise=true , address=77.90.31.16:4758 , piece_id=THMBUGGESSBITSVHQ37Z2QOOY7PWR5IJZD35OCIFR5O7RHEAEWOA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.728248987 (128h19m44.555957387s) [noise=true , address=145.79.227.21:4486 , piece_id=ZN26TECSR257S245ZX2L7QJLHEMTYDFKRAEKSF5HW3D6ZZ4KXO4Q, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.728690766 (128h19m44.556399164s) [noise=true , address=45.91.116.9:4265 , piece_id=MASTGPEUEXCCLO62MPUIY6HBKXC4PIGSLJX4JEDXYGHF6RLSVVKA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.729206478 (128h19m44.556914879s) [noise=true , address=145.79.227.31:4829 , piece_id=JXWYGENJKBFEOA3DMV6UMKYTRT3X62KIIILBERQXHFPVJGHZXIOA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.729563575 (128h19m44.55727197s) [noise=true , address=145.79.227.40:4064 , piece_id=RRQVPNCTREHCHLQZXEYKFYMYWQGFFROBQIV45ZQXJXFHTQBTJPFQ, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.729872439 (128h19m44.557580831s) [noise=true , address=45.91.116.11:4135 , piece_id=UTN55QJTH622ZLEV6KGPSYJLGP4XC5MVHOX2SCH2GCBITD7ZZYQA, seg_pos={0 0}, upload=133438]: Uploading piece...
Dec 10 23:47:43.794956612 (128h19m44.622664993s) [upload=133438]: Next returned nil segment
Dec 10 23:47:43.794973774 (128h19m44.622682178s) [upload=133438]: Waiting for error group managing segments...
Dec 10 23:47:43.800127388 (128h19m44.627835779s) [noise=true , address=45.91.116.9:4265 , piece_id=MASTGPEUEXCCLO62MPUIY6HBKXC4PIGSLJX4JEDXYGHF6RLSVVKA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.800237327 (128h19m44.627945718s) [noise=true , address=45.91.116.10:4104 , piece_id=UTYNQK3VJTGSB6RB5GFDWCL4FCID6KKONMN7ABTVAY6LWJ6HIXMA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.801134286 (128h19m44.62884268s) [noise=true , address=45.91.116.13:4181 , piece_id=PES4XRWCRUAAGUCW32A4ZW3ZRJ6JQWPU4G6E7ZXS6XQFWWEI7HSQ, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.801669629 (128h19m44.62937802s) [noise=true , address=145.79.227.31:4829 , piece_id=JXWYGENJKBFEOA3DMV6UMKYTRT3X62KIIILBERQXHFPVJGHZXIOA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.805790521 (128h19m44.633498915s) [noise=true , address=145.79.227.33:4893 , piece_id=2GTGRATIIXJ53SFI3CKRSMEMQOYEGPKA4ODXQZE6XDUXIMMGSW2A, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.815227011 (128h19m44.642935412s) [noise=true , address=45.91.116.14:4195 , piece_id=5GDMRZS2YNFQTGQXVE2XG3EEJ2PORUXQATCNWMSJE26JYHRBHA7A, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.816041314 (128h19m44.64374972s) [noise=true , address=45.91.116.11:4135 , piece_id=UTN55QJTH622ZLEV6KGPSYJLGP4XC5MVHOX2SCH2GCBITD7ZZYQA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.819850848 (128h19m44.64755924s) [noise=true , address=145.79.227.40:4064 , piece_id=RRQVPNCTREHCHLQZXEYKFYMYWQGFFROBQIV45ZQXJXFHTQBTJPFQ, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.820406780 (128h19m44.648115179s) [noise=true , address=145.79.227.21:4486 , piece_id=ZN26TECSR257S245ZX2L7QJLHEMTYDFKRAEKSF5HW3D6ZZ4KXO4Q, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.822071611 (128h19m44.649780002s) [noise=true , address=45.91.116.7:4025 , piece_id=Y4WVBL3ZMWLIB4U5OEY4WZZVS3THRFE7THC5ZMO73TZK7PGA6NCA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.830587944 (128h19m44.658296338s) [noise=true , address=145.79.227.2:4126 , piece_id=5NWC43ZT64OTLF7I3OJTAE5C6ZKWSKPQMLZY3FIB4ADNIEGFEJ7A, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:43.883733278 (128h19m44.711441669s) [noise=true , address=5.231.73.4:4028 , piece_id=FMUEVCR4LJFLC3QO6C4DWERVZUWR3MX4NQ3BUFSFQRUWWALPKNWA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:47:44.655889024 (128h19m45.48359742s) [noise=true , address=145.79.227.16:4935 , piece_id=UI2JPSHBXXKBSRLQ3LCZH2K6JX54N5XBYJA7HCMCTL27URJHKIYQ, seg_pos={0 0}, upload=133438]: Done uploading piece. err: <nil>
Dec 10 23:48:43.647990068 (128h20m44.475698446s) [noise=true , address=77.90.31.18:4822 , piece_id=VKGXIPMO53OFOHZZDR6AZIW7UOSQMOXO3DCWU63IQTJZTMMRRMRQ, seg_pos={0 0}, upload=133438]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: write tcp 159.223.143.39:50220->77.90.31.18:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:43.648057376 (128h20m44.475765778s) [noise=true , address=77.90.31.17:4787 , piece_id=B47TMDN4VCVUPUJHI7R7GBEHY3DPIXDV4RMN33ULB7S5R66K5YFA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:43.648165933 (128h20m44.475874333s) [noise=true , address=77.90.31.14:4683 , piece_id=TL5IIMRX7SLBH77ZD5OLVIZ57VG455PK426NATNUBXOL746JBMMQ, seg_pos={0 0}, upload=133438]: Done uploading piece. err: ecclient: upload failed (node:12cth5bdi5iAWhHcLxqN9c7GLdheS7SLQ3x5fEHG1MdgdXWFdD6, address:77.90.31.14:4683): protocol: rpc: dial tcp 77.90.31.14:4683: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:43.648280867 (128h20m44.475989265s) [noise=true , address=77.90.31.13:4660 , piece_id=IUFYQGDXWVX5ST77OZ3SPKYXOD72CYW5IX42TBXQOLX3DTAI7J2A, seg_pos={0 0}, upload=133438]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: rpc: dial tcp 77.90.31.13:4660: i/o timeout; conn closed: op
Dec 10 23:48:43.648337189 (128h20m44.476045584s) [noise=true , address=77.90.31.16:4758 , piece_id=THMBUGGESSBITSVHQ37Z2QOOY7PWR5IJZD35OCIFR5O7RHEAEWOA, seg_pos={0 0}, upload=133438]: Done uploading piece. err: ecclient: upload failed (node:1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, address:77.90.31.16:4758): protocol: write tcp 159.223.143.39:35528->77.90.31.16:4758: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:43.648368501 (128h20m44.476076878s) [seg_pos={0 0}, upload=133438]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 10 23:48:43.671512351 (128h20m44.499220744s) [upload=133438]: Done uploading segments...
Dec 10 23:48:43.671553096 (128h20m44.499261501s) [upload=133438]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.0251551s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........................XXX.X.............................] duration:
446.554947ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:48:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 446.653934ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
21 out of 60 is failed (threshold=3) [...................................XX..XXXXXX.XXXXXX.XXXXXXX] duration:
1m0.025885666s trace:
https://jaeger.dev.storj.io/trace/2ba770733e6b489f
FAIL (2025-12-10 23:48:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:47:27.601941971 (128h19m28.429650367s) [upload=133434]: Starting upload
Dec 10 23:47:27.601990830 (128h19m28.429699221s) [upload=133434]: Done starting upload
Dec 10 23:47:27.602262753 (128h19m28.429971144s) [upload=133434]: Uploading segments...
Dec 10 23:47:27.602302793 (128h19m28.43001119s) [upload=133434]: Got next segment. Inline: false
Dec 10 23:47:27.602333189 (128h19m28.43004158s) [upload=133434]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:47:27.602337249 (128h19m28.430045639s) [upload=133434]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:47:27.641977390 (128h19m28.469685781s) [seg_pos={0 0}, upload=133434]: Begin upload segment...
Dec 10 23:47:27.642047687 (128h19m28.46975608s) [seg_pos={0 0}, upload=133434]: Done begin upload segment.
Dec 10 23:47:27.642084049 (128h19m28.469792446s) [noise=true , address=145.79.227.34:4927 , piece_id=QIHDK3SAOHLHJXAZ4I6I23TABNZHQPBLHHOP4REFEFHPLNVGEXGA, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.643222876 (128h19m28.470931269s) [noise=true , address=45.91.116.6:4990 , piece_id=6BSQNL6Z5CEZRCL4ONOV5RFLBP3GFTSDLEW4QADB72WJQFC6WX7A, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.643250553 (128h19m28.470958947s) [noise=true , address=77.90.31.18:4822 , piece_id=7RV3EOIAOAWNJS5G6VQ7YPG3Y452KFAIDQ5ENTC4HXTZX73WKGAA, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.643390406 (128h19m28.471098803s) [noise=true , address=145.79.227.27:4696 , piece_id=5AJNGVXZUXHNZL646ASZ2FEKK6IKCQT35WYMIZNTJPZ6EJXH4SOA, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.643759231 (128h19m28.471467627s) [noise=true , address=77.90.31.21:4882 , piece_id=CYHWQXCO6EOOAJ4CSPOLDKULDOV75UHIZS5AXOA7TR5UMHOA7UMQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.643839687 (128h19m28.471548081s) [noise=true , address=77.90.31.13:4656 , piece_id=XY7ZTCYQFTSGRDGFXKMI4YJHYLSOYGBM7IWCTR36KOM7C5R27MZQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.644208478 (128h19m28.471916898s) [noise=true , address=77.90.31.15:4725 , piece_id=PXXXYFXJ4472ZPFUI5VNDTQOTEZJH2QJJNNOIS2FQYPCKMYWIQ5Q, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.644221441 (128h19m28.471929837s) [noise=true , address=45.91.116.12:4240 , piece_id=R4E27ZYKZGAICA47YRO7HHTUNHW376DLA3ZXPFIQ4WAOXS4R37KQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.644879122 (128h19m28.472587524s) [noise=true , address=77.90.31.17:4790 , piece_id=TPWKO4DGHGKRTRNW5GQY2C7TQHQ2Y77JLA5F5ISRGNUKOQO3DNZQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.644926128 (128h19m28.472634528s) [noise=true , address=145.79.227.32:4839 , piece_id=HJDXFMHRWJ7J7QTZSAL76MFY4F6GPQSWDQRO5KOBXNADBNIYKTZA, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.645244155 (128h19m28.472952552s) [noise=true , address=45.91.116.11:4151 , piece_id=OKAIEKQZLSHN7CXL2DSJYGZC24RQKHKF3DLY7POSH3J2FJNMA5IA, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.645343380 (128h19m28.473051771s) [noise=true , address=5.231.73.2:4007 , piece_id=BT3YWU4ZZFV7YFEQIGZ7MQ6L5OEGNZHKF57YMDIUC43VQLUFYS6Q, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.645570319 (128h19m28.473278716s) [noise=true , address=77.90.31.12:4605 , piece_id=LQPEPSO6QPD4I5EFOUU5GW2ZKJL36YNNSA64PMKPQRE6RG6KV77A, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.645889715 (128h19m28.473598115s) [noise=true , address=45.91.116.9:4264 , piece_id=OZ6IYNRZVGER6752FAVSBUV3IHGPAL6EC2LDBDTXGITMB7T3SWDQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.646202617 (128h19m28.473911008s) [noise=true , address=145.79.227.30:4772 , piece_id=VKOHCGYACHBPFB52WIAH53ADZAHFDVY4WYISF74L7KUVPZ4OYZ5A, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.646558221 (128h19m28.474266615s) [noise=true , address=45.91.116.7:4028 , piece_id=IN5LBEVSBOCSED3DYYUERY6EQZ546G5RYGTMTR6XJC533LYEEUJA, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.650753466 (128h19m28.478461866s) [noise=true , address=145.79.227.29:4968 , piece_id=PYWRWHKKHZBP6YDGWVAVVACXBL2NZZ33B654C4H5YXH6ZKE6SZOQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.652651569 (128h19m28.480359954s) [noise=true , address=45.91.116.5:4327 , piece_id=AB5YFDAHMPSCODQEFPPVUC37CHJ4MP5YBKUZEYFOKIADBXN4O3XQ, seg_pos={0 0}, upload=133434]: Uploading piece...
Dec 10 23:47:27.710367983 (128h19m28.53807636s) [upload=133434]: Next returned nil segment
Dec 10 23:47:27.710380848 (128h19m28.538089243s) [upload=133434]: Waiting for error group managing segments...
Dec 10 23:47:27.714332109 (128h19m28.542040486s) [noise=true , address=45.91.116.6:4990 , piece_id=6BSQNL6Z5CEZRCL4ONOV5RFLBP3GFTSDLEW4QADB72WJQFC6WX7A, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.714762901 (128h19m28.542471292s) [noise=true , address=45.91.116.12:4240 , piece_id=R4E27ZYKZGAICA47YRO7HHTUNHW376DLA3ZXPFIQ4WAOXS4R37KQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.719119395 (128h19m28.546827787s) [noise=true , address=145.79.227.34:4927 , piece_id=QIHDK3SAOHLHJXAZ4I6I23TABNZHQPBLHHOP4REFEFHPLNVGEXGA, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.721416543 (128h19m28.54912492s) [noise=true , address=145.79.227.27:4696 , piece_id=5AJNGVXZUXHNZL646ASZ2FEKK6IKCQT35WYMIZNTJPZ6EJXH4SOA, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.725915804 (128h19m28.553624187s) [noise=true , address=45.91.116.11:4151 , piece_id=OKAIEKQZLSHN7CXL2DSJYGZC24RQKHKF3DLY7POSH3J2FJNMA5IA, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.725945563 (128h19m28.553653946s) [noise=true , address=145.79.227.32:4839 , piece_id=HJDXFMHRWJ7J7QTZSAL76MFY4F6GPQSWDQRO5KOBXNADBNIYKTZA, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.731695806 (128h19m28.559404205s) [noise=true , address=45.91.116.7:4028 , piece_id=IN5LBEVSBOCSED3DYYUERY6EQZ546G5RYGTMTR6XJC533LYEEUJA, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.732435508 (128h19m28.5601439s) [noise=true , address=45.91.116.5:4327 , piece_id=AB5YFDAHMPSCODQEFPPVUC37CHJ4MP5YBKUZEYFOKIADBXN4O3XQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.734530902 (128h19m28.562239298s) [noise=true , address=45.91.116.9:4264 , piece_id=OZ6IYNRZVGER6752FAVSBUV3IHGPAL6EC2LDBDTXGITMB7T3SWDQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:27.800217489 (128h19m28.62792588s) [noise=true , address=145.79.227.29:4968 , piece_id=PYWRWHKKHZBP6YDGWVAVVACXBL2NZZ33B654C4H5YXH6ZKE6SZOQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:28.715424904 (128h19m29.543133296s) [noise=true , address=145.79.227.30:4772 , piece_id=VKOHCGYACHBPFB52WIAH53ADZAHFDVY4WYISF74L7KUVPZ4OYZ5A, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:47:28.808498877 (128h19m29.636207275s) [noise=true , address=5.231.73.2:4007 , piece_id=BT3YWU4ZZFV7YFEQIGZ7MQ6L5OEGNZHKF57YMDIUC43VQLUFYS6Q, seg_pos={0 0}, upload=133434]: Done uploading piece. err: <nil>
Dec 10 23:48:27.573607577 (128h20m28.40131598s) [noise=true , address=77.90.31.15:4725 , piece_id=PXXXYFXJ4472ZPFUI5VNDTQOTEZJH2QJJNNOIS2FQYPCKMYWIQ5Q, seg_pos={0 0}, upload=133434]: Done uploading piece. err: ecclient: upload failed (node:1nstPr2GMJo6pnBmBuDME5QYw3mXgnqLoaTKkcKUhjSbAY8mQk, address:77.90.31.15:4725): protocol: write tcp 159.223.143.39:58528->77.90.31.15:4725: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:27.574078151 (128h20m28.401786552s) [noise=true , address=77.90.31.13:4656 , piece_id=XY7ZTCYQFTSGRDGFXKMI4YJHYLSOYGBM7IWCTR36KOM7C5R27MZQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: write tcp 159.223.143.39:51400->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:27.574125587 (128h20m28.401833983s) [noise=true , address=77.90.31.17:4790 , piece_id=TPWKO4DGHGKRTRNW5GQY2C7TQHQ2Y77JLA5F5ISRGNUKOQO3DNZQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: write tcp 159.223.143.39:49592->77.90.31.17:4790: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:27.574562736 (128h20m28.402271134s) [noise=true , address=77.90.31.12:4605 , piece_id=LQPEPSO6QPD4I5EFOUU5GW2ZKJL36YNNSA64PMKPQRE6RG6KV77A, seg_pos={0 0}, upload=133434]: Done uploading piece. err: ecclient: upload failed (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53, address:77.90.31.12:4605): protocol: write tcp 159.223.143.39:52184->77.90.31.12:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:27.574605920 (128h20m28.402314321s) [noise=true , address=77.90.31.18:4822 , piece_id=7RV3EOIAOAWNJS5G6VQ7YPG3Y452KFAIDQ5ENTC4HXTZX73WKGAA, seg_pos={0 0}, upload=133434]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: write tcp 159.223.143.39:53322->77.90.31.18:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:27.574888773 (128h20m28.402597174s) [noise=true , address=77.90.31.21:4882 , piece_id=CYHWQXCO6EOOAJ4CSPOLDKULDOV75UHIZS5AXOA7TR5UMHOA7UMQ, seg_pos={0 0}, upload=133434]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:33348->77.90.31.21:4882: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:48:27.575685075 (128h20m28.403393453s) [seg_pos={0 0}, upload=133434]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:48:27.596487509 (128h20m28.424195903s) [upload=133434]: Done uploading segments...
Dec 10 23:48:27.596710039 (128h20m28.424418437s) [upload=133434]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.025901867s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................XXX................................] duration:
545.556926ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:48:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 545.591581ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................X....X...X........X...X..............] duration:
429.342906ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:47:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 429.381387ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [....................................XXXXXXX.XXXXXXXXXXX.XXXX] duration:
1m0.020125986s trace:
https://jaeger.dev.storj.io/trace/f40887b8876ec96
FAIL (2025-12-10 23:47:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
18m44.436604359s) [upload=133421]: Uploading segments...
Dec 10 23:46:43.609205858 (128h18m44.436914249s) [upload=133421]: Got next segment. Inline: false
Dec 10 23:46:43.609252119 (128h18m44.43696051s) [upload=133421]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:46:43.609257988 (128h18m44.436966379s) [upload=133421]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:46:43.635102359 (128h18m44.462810762s) [seg_pos={0 0}, upload=133421]: Begin upload segment...
Dec 10 23:46:43.635178001 (128h18m44.462886397s) [seg_pos={0 0}, upload=133421]: Done begin upload segment.
Dec 10 23:46:43.635209713 (128h18m44.462918116s) [noise=true , address=77.90.31.21:4880 , piece_id=RDGQ52DCGNWGASQKBT3IW2TDO6D3QJZRMUXK72IZXE4NNTTPTPTA, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.635231396 (128h18m44.462939793s) [noise=true , address=77.90.31.18:4810 , piece_id=KOID2CLXM5B6TKVUX6PTMJZ25KM5QBAH6MZDNRY2JSMPWOX6WJ5Q, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.635389936 (128h18m44.463098337s) [noise=true , address=5.231.73.2:4019 , piece_id=2J3UYIBIKABXKZQZCT3TBL6T7NS6BN7BQ37ISLIKHRXFVGZA2JUQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.635910239 (128h18m44.46361864s) [noise=true , address=145.79.227.30:4765 , piece_id=RWY2WPKCCGGPAFG5C3COLDTQQAHR2FJBR4BXO5UOEHT3W6KJITMQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.636500322 (128h18m44.464208723s) [noise=true , address=5.231.73.4:4005 , piece_id=UR2NHLQU6XZZJWXRUTGZHKU6HXDUHH6MDT5DOE4ZLJ6KWANHKHOA, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.637197830 (128h18m44.464906233s) [noise=true , address=145.79.227.14:4866 , piece_id=AJWXCYXND56CCNSKZ6H5DN7YAH4QMSXHFZQA3XEYUOFPQMFIRZVQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.637804698 (128h18m44.4655131s) [noise=true , address=45.91.116.14:4210 , piece_id=6IVCRQXDTZBODRC7QDUEEJC37COYLKPY7Z27Q4BPKGDPGBJCECRQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.638379118 (128h18m44.466087518s) [noise=true , address=145.79.227.42:4129 , piece_id=ZQF4MBQ6DBTZYGSQPFJC55CTZU6NFGP5CP76MMVGHGGDU76ADKGQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.638908365 (128h18m44.466616771s) [noise=true , address=145.79.227.49:4344 , piece_id=RN6F7WSQITEAR7CKKZILMEB4ARFGR2EUMPKQGBI2KOWMOWBL2VEQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.639826218 (128h18m44.467534619s) [noise=true , address=45.91.116.14:4201 , piece_id=K2YEPZVY5ENQRMVV743PG4F7SL2JHQJVBEARSMONQ3SS2BHHXV6A, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.640486900 (128h18m44.468195303s) [noise=true , address=77.90.31.19:4875 , piece_id=KZRSRO7FQUITG4E4UWCYOBQLWKB7FUK7IQR6YKMQLJGR2LXOWXIA, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.641054070 (128h18m44.468762467s) [noise=true , address=77.90.31.12:4620 , piece_id=3YKCBQE5RDEZZXOVJH267SI2CWFQN7HS4ZK6ZFJDBOMF5SMY2VPQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.641173570 (128h18m44.46888197s) [noise=true , address=145.79.227.28:4746 , piece_id=WFNDO57H3B7SANX7RYQXADTJYCGREWT4OGKRDJURBZBSRMUX2VDA, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.641745899 (128h18m44.469454301s) [noise=true , address=77.90.31.10:4528 , piece_id=NPYYXD3YBHRF52X43OFN2QZABYRWTJT6THDDUU6CYNHLFEO4IPXQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.642295394 (128h18m44.470003796s) [noise=true , address=145.79.227.45:4233 , piece_id=KTLGSARBBLN5S4U35GPSGJMVFZ6DGJZYQYCRTUDOJTTKUAIU76YA, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.643051905 (128h18m44.470760306s) [noise=true , address=64.49.13.169:4716 , piece_id=HTJXIKYXUCKXI54XRFBRGGKH422PL4FCVKJ7EHTEHFWM57RXGBYQ, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.643068512 (128h18m44.470776913s) [noise=true , address=145.79.227.16:4926 , piece_id=FJIHLNWV27OTSTHVAOR6WUHTEDD2C36NRIDSHHMY4A4SLMODUU7A, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.643797964 (128h18m44.471506371s) [noise=true , address=45.91.116.10:4115 , piece_id=AXUDCXSYKS5WKUWSP3EW4EX7YZHQ7SLDFPQE3RDGKESHLWFE4MRA, seg_pos={0 0}, upload=133421]: Uploading piece...
Dec 10 23:46:43.719778969 (128h18m44.547487376s) [upload=133421]: Next returned nil segment
Dec 10 23:46:43.719796189 (128h18m44.547504589s) [upload=133421]: Waiting for error group managing segments...
Dec 10 23:46:43.724137784 (128h18m44.551846179s) [noise=true , address=45.91.116.10:4115 , piece_id=AXUDCXSYKS5WKUWSP3EW4EX7YZHQ7SLDFPQE3RDGKESHLWFE4MRA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.726077753 (128h18m44.553786155s) [noise=true , address=45.91.116.14:4201 , piece_id=K2YEPZVY5ENQRMVV743PG4F7SL2JHQJVBEARSMONQ3SS2BHHXV6A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.739900740 (128h18m44.567609135s) [noise=true , address=145.79.227.28:4746 , piece_id=WFNDO57H3B7SANX7RYQXADTJYCGREWT4OGKRDJURBZBSRMUX2VDA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.743792756 (128h18m44.57150115s) [noise=true , address=45.91.116.14:4210 , piece_id=6IVCRQXDTZBODRC7QDUEEJC37COYLKPY7Z27Q4BPKGDPGBJCECRQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.744334314 (128h18m44.572042714s) [noise=true , address=145.79.227.14:4866 , piece_id=AJWXCYXND56CCNSKZ6H5DN7YAH4QMSXHFZQA3XEYUOFPQMFIRZVQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.747288531 (128h18m44.574996933s) [noise=true , address=145.79.227.45:4233 , piece_id=KTLGSARBBLN5S4U35GPSGJMVFZ6DGJZYQYCRTUDOJTTKUAIU76YA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.762723207 (128h18m44.590431598s) [noise=true , address=145.79.227.42:4129 , piece_id=ZQF4MBQ6DBTZYGSQPFJC55CTZU6NFGP5CP76MMVGHGGDU76ADKGQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.796532161 (128h18m44.624240556s) [noise=true , address=5.231.73.4:4005 , piece_id=UR2NHLQU6XZZJWXRUTGZHKU6HXDUHH6MDT5DOE4ZLJ6KWANHKHOA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.827369732 (128h18m44.655078124s) [noise=true , address=5.231.73.2:4019 , piece_id=2J3UYIBIKABXKZQZCT3TBL6T7NS6BN7BQ37ISLIKHRXFVGZA2JUQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:43.902135549 (128h18m44.729843996s) [noise=true , address=145.79.227.49:4344 , piece_id=RN6F7WSQITEAR7CKKZILMEB4ARFGR2EUMPKQGBI2KOWMOWBL2VEQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:44.695507682 (128h18m45.523216064s) [noise=true , address=145.79.227.30:4765 , piece_id=RWY2WPKCCGGPAFG5C3COLDTQQAHR2FJBR4BXO5UOEHT3W6KJITMQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:44.748180623 (128h18m45.575889017s) [noise=true , address=145.79.227.16:4926 , piece_id=FJIHLNWV27OTSTHVAOR6WUHTEDD2C36NRIDSHHMY4A4SLMODUU7A, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:46:50.207578551 (128h18m51.035286944s) [noise=true , address=64.49.13.169:4716 , piece_id=HTJXIKYXUCKXI54XRFBRGGKH422PL4FCVKJ7EHTEHFWM57RXGBYQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: <nil>
Dec 10 23:47:43.564535004 (128h19m44.392243416s) [noise=true , address=77.90.31.21:4880 , piece_id=RDGQ52DCGNWGASQKBT3IW2TDO6D3QJZRMUXK72IZXE4NNTTPTPTA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, address:77.90.31.21:4880): protocol: rpc: dial tcp 77.90.31.21:4880: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:43.564729888 (128h19m44.392438289s) [noise=true , address=77.90.31.12:4620 , piece_id=3YKCBQE5RDEZZXOVJH267SI2CWFQN7HS4ZK6ZFJDBOMF5SMY2VPQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: rpc: dial tcp 77.90.31.12:4620: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:43.564988581 (128h19m44.392696976s) [noise=true , address=77.90.31.19:4875 , piece_id=KZRSRO7FQUITG4E4UWCYOBQLWKB7FUK7IQR6YKMQLJGR2LXOWXIA, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:1xSFQaW3aVnAVDYE5r5ADmCGU95vWnshuZyte32W2YAj94avNn, address:77.90.31.19:4875): protocol: rpc: dial tcp 77.90.31.19:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:43.565242395 (128h19m44.392950795s) [noise=true , address=77.90.31.10:4528 , piece_id=NPYYXD3YBHRF52X43OFN2QZABYRWTJT6THDDUU6CYNHLFEO4IPXQ, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:15fkvAWei1E5Tn2BYkCavyZ8se6XwPJcmpr7aEsv8sbMtk1n3Q, address:77.90.31.10:4528): protocol: rpc: dial tcp 77.90.31.10:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:43.565367532 (128h19m44.393075927s) [noise=true , address=77.90.31.18:4810 , piece_id=KOID2CLXM5B6TKVUX6PTMJZ25KM5QBAH6MZDNRY2JSMPWOX6WJ5Q, seg_pos={0 0}, upload=133421]: Done uploading piece. err: ecclient: upload failed (node:12mfG3ZeKyFWEPsTgSoR4wqiUYY7YobwmMtsg1GsCorquCZLX5F, address:77.90.31.18:4810): protocol: rpc: dial tcp 77.90.31.18:4810: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:43.565404484 (128h19m44.393112861s) [seg_pos={0 0}, upload=133421]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 10 23:47:43.582897934 (128h19m44.410606325s) [upload=133421]: Done uploading segments...
Dec 10 23:47:43.582932270 (128h19m44.410640663s) [upload=133421]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.020139405s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........................XXX.X............................] duration:
414.849146ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:47:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 414.873015ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................X....X...X........X...X.............] duration:
455.286563ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:47:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 455.318339ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration:
515.676033ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:47:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 515.726483ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
20 out of 60 is failed (threshold=3) [....................................XX..XXXXXX.XXXXXX.XXXXXX] duration:
1m0.023450259s trace:
https://jaeger.dev.storj.io/trace/528e5aa244de56a3
FAIL (2025-12-10 23:47:27 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:46:27.521772255 (128h18m28.349480647s) [upload=133414]: Starting upload
Dec 10 23:46:27.521826533 (128h18m28.349534914s) [upload=133414]: Done starting upload
Dec 10 23:46:27.521929909 (128h18m28.3496383s) [upload=133414]: Uploading segments...
Dec 10 23:46:27.522084484 (128h18m28.349792877s) [upload=133414]: Got next segment. Inline: false
Dec 10 23:46:27.522125195 (128h18m28.34983359s) [upload=133414]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:46:27.522130787 (128h18m28.349839182s) [upload=133414]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:46:27.552288965 (128h18m28.37999736s) [seg_pos={0 0}, upload=133414]: Begin upload segment...
Dec 10 23:46:27.552351428 (128h18m28.380059812s) [seg_pos={0 0}, upload=133414]: Done begin upload segment.
Dec 10 23:46:27.552396385 (128h18m28.380104785s) [noise=true , address=145.79.227.17:4979 , piece_id=Q5FSXKYQHECZA3IX6R7BXETUPHXARWVYPBMMLGUEUXOUZPASDS3A, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.552412945 (128h18m28.380121347s) [noise=true , address=145.79.227.33:4878 , piece_id=YEYYGER5FKCVFOB5AIJTYWCLXU6Z2QLF56DLGAWW2XNO5PK2M23Q, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.553071671 (128h18m28.380780062s) [noise=true , address=77.90.31.12:4613 , piece_id=ERGWX34BTOPA7MLB3VBA7QT74PXC5CYEVLOV5M6ANHIGKGBKJN4Q, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.553453440 (128h18m28.381161832s) [noise=true , address=77.90.31.21:4895 , piece_id=FWVIS22YGX2DNSQZ4QVHKNPTJEFYCYBQ5WOFNG4JSOZSHIUVN3QQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.553836126 (128h18m28.381544517s) [noise=true , address=77.90.31.16:4754 , piece_id=ZUGDZXDFKYH62HYCIPG34ZHNXF7GY5OHSBMOQ6QGGM2R2GTEFJUQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.554204715 (128h18m28.381913106s) [noise=true , address=64.49.13.169:4711 , piece_id=PUO2QKNBA6OTPS2SD5QXFJ5SQICZBEFTGTJDDUOJS2CR5YTMKA3A, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.554759602 (128h18m28.382468002s) [noise=true , address=77.90.31.14:4696 , piece_id=NSHCRJY4NSZIFN3XYHVNFGXTHAOWH7ZFS2RJ2RXW7COYPWPNINLQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.555337311 (128h18m28.383045708s) [noise=true , address=145.79.227.34:4912 , piece_id=T5KFZ2HLFCHLH4T5WLRZ65TE2YWDCD6EUTL2O3YYVCVHEBTWQIPQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.555798757 (128h18m28.383507154s) [noise=true , address=145.79.227.35:4491 , piece_id=U75FEMK6Z7D7MVIKYDACJCINANDA5QCY7XB5M7ZX6KR4YYD4AQZQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.556389218 (128h18m28.38409762s) [noise=true , address=5.231.73.4:4019 , piece_id=EUJEXLOE3XYEJQDLLUMWFHLYZIFTBLIXPNZLVDWX64DLBA27APZQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.557181643 (128h18m28.384890046s) [noise=true , address=45.91.116.4:4951 , piece_id=XHYS5XZH563DKSTWUUB2CS4DCUSEVQBZWI5ZBADRAXFD5CTKUXHA, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.557698861 (128h18m28.385407262s) [noise=true , address=77.90.31.11:4588 , piece_id=62BRWZHEVVTR7HMYFF3373YOPYXRQWYW7XXKFTGVYDSYUK6LRSUA, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.558498013 (128h18m28.386206404s) [noise=true , address=45.91.116.8:4061 , piece_id=BJTVO6UVYYSPDAYDPEEXBGEPYBEUYQQCJEPQCBFGJ4G7GJAWPUDQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.558727512 (128h18m28.386435904s) [noise=true , address=145.79.227.2:4136 , piece_id=CIFTE3C2OABX76G52EKV7LIE7QRIRVE77NGOW5TVESBJAD6VL56Q, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.559482548 (128h18m28.387190946s) [noise=true , address=145.79.227.31:4813 , piece_id=LNJLUPCGG5L6U6WDWPW5TCP67XQHDEWWZ4MX73CGQX27MF7VKIBQ, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.559779680 (128h18m28.387488077s) [noise=true , address=145.79.227.39:4026 , piece_id=4ZUWCAOIR6OQWCOSLVQ4F4YK4XMBP5VZJJ2EB3BJIDEY4HGXPCUA, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.560177914 (128h18m28.387886308s) [noise=true , address=77.90.31.13:4657 , piece_id=4TON6A2AAPELWDVZBIG26CFPQ43S6OCRE4XZPZA5VA6V7AYUA2RA, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.562028083 (128h18m28.389736474s) [noise=true , address=45.91.116.9:4268 , piece_id=EPYB4NXQ4U3MYFT4BZVA5ZRRWJZBH6MRWVTQB7XLXV45JKW5TA6A, seg_pos={0 0}, upload=133414]: Uploading piece...
Dec 10 23:46:27.620522681 (128h18m28.448231073s) [upload=133414]: Next returned nil segment
Dec 10 23:46:27.620535911 (128h18m28.448244305s) [upload=133414]: Waiting for error group managing segments...
Dec 10 23:46:27.627101777 (128h18m28.454810168s) [noise=true , address=145.79.227.39:4026 , piece_id=4ZUWCAOIR6OQWCOSLVQ4F4YK4XMBP5VZJJ2EB3BJIDEY4HGXPCUA, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.627325468 (128h18m28.455033859s) [noise=true , address=145.79.227.17:4979 , piece_id=Q5FSXKYQHECZA3IX6R7BXETUPHXARWVYPBMMLGUEUXOUZPASDS3A, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.628670330 (128h18m28.456378722s) [noise=true , address=145.79.227.34:4912 , piece_id=T5KFZ2HLFCHLH4T5WLRZ65TE2YWDCD6EUTL2O3YYVCVHEBTWQIPQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.636886631 (128h18m28.464595027s) [noise=true , address=45.91.116.8:4061 , piece_id=BJTVO6UVYYSPDAYDPEEXBGEPYBEUYQQCJEPQCBFGJ4G7GJAWPUDQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.639430692 (128h18m28.467139084s) [noise=true , address=45.91.116.4:4951 , piece_id=XHYS5XZH563DKSTWUUB2CS4DCUSEVQBZWI5ZBADRAXFD5CTKUXHA, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.640005702 (128h18m28.467714096s) [noise=true , address=145.79.227.33:4878 , piece_id=YEYYGER5FKCVFOB5AIJTYWCLXU6Z2QLF56DLGAWW2XNO5PK2M23Q, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.644781482 (128h18m28.472489873s) [noise=true , address=45.91.116.9:4268 , piece_id=EPYB4NXQ4U3MYFT4BZVA5ZRRWJZBH6MRWVTQB7XLXV45JKW5TA6A, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.645921956 (128h18m28.473630347s) [noise=true , address=145.79.227.2:4136 , piece_id=CIFTE3C2OABX76G52EKV7LIE7QRIRVE77NGOW5TVESBJAD6VL56Q, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.653404241 (128h18m28.481112631s) [noise=true , address=145.79.227.31:4813 , piece_id=LNJLUPCGG5L6U6WDWPW5TCP67XQHDEWWZ4MX73CGQX27MF7VKIBQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.678001978 (128h18m28.50571037s) [noise=true , address=145.79.227.35:4491 , piece_id=U75FEMK6Z7D7MVIKYDACJCINANDA5QCY7XB5M7ZX6KR4YYD4AQZQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:27.720293712 (128h18m28.548002103s) [noise=true , address=5.231.73.4:4019 , piece_id=EUJEXLOE3XYEJQDLLUMWFHLYZIFTBLIXPNZLVDWX64DLBA27APZQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:46:33.580675327 (128h18m34.408383721s) [noise=true , address=64.49.13.169:4711 , piece_id=PUO2QKNBA6OTPS2SD5QXFJ5SQICZBEFTGTJDDUOJS2CR5YTMKA3A, seg_pos={0 0}, upload=133414]: Done uploading piece. err: <nil>
Dec 10 23:47:27.487189226 (128h19m28.314897624s) [noise=true , address=77.90.31.21:4895 , piece_id=FWVIS22YGX2DNSQZ4QVHKNPTJEFYCYBQ5WOFNG4JSOZSHIUVN3QQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: ecclient: upload failed (node:12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, address:77.90.31.21:4895): protocol: rpc: dial tcp 77.90.31.21:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:27.487375676 (128h19m28.315084074s) [noise=true , address=77.90.31.14:4696 , piece_id=NSHCRJY4NSZIFN3XYHVNFGXTHAOWH7ZFS2RJ2RXW7COYPWPNINLQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:52304->77.90.31.14:4696: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:27.487444239 (128h19m28.315152636s) [noise=true , address=77.90.31.11:4588 , piece_id=62BRWZHEVVTR7HMYFF3373YOPYXRQWYW7XXKFTGVYDSYUK6LRSUA, seg_pos={0 0}, upload=133414]: Done uploading piece. err: ecclient: upload failed (node:12H6EaBk5vQxDeccjDtyoJmuANkspKuRPCyS7qHuDn16GQPa8o6, address:77.90.31.11:4588): protocol: rpc: dial tcp 77.90.31.11:4588: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:27.487485227 (128h19m28.31519361s) [noise=true , address=77.90.31.16:4754 , piece_id=ZUGDZXDFKYH62HYCIPG34ZHNXF7GY5OHSBMOQ6QGGM2R2GTEFJUQ, seg_pos={0 0}, upload=133414]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:27.487590322 (128h19m28.315298719s) [noise=true , address=77.90.31.13:4657 , piece_id=4TON6A2AAPELWDVZBIG26CFPQ43S6OCRE4XZPZA5VA6V7AYUA2RA, seg_pos={0 0}, upload=133414]: Done uploading piece. err: ecclient: upload failed (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt, address:77.90.31.13:4657): protocol: write tcp 159.223.143.39:56296->77.90.31.13:4657: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:27.487659994 (128h19m28.31536839s) [noise=true , address=77.90.31.12:4613 , piece_id=ERGWX34BTOPA7MLB3VBA7QT74PXC5CYEVLOV5M6ANHIGKGBKJN4Q, seg_pos={0 0}, upload=133414]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: rpc: dial tcp 77.90.31.12:4613: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:47:27.487686720 (128h19m28.315395104s) [seg_pos={0 0}, upload=133414]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:47:27.508932225 (128h19m28.336640619s) [upload=133414]: Done uploading segments...
Dec 10 23:47:27.508978153 (128h19m28.33668655s) [upload=133414]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.023473033s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................X....X...X........X...X............] duration:
471.74284ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:47:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.768473ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration:
554.128816ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:47:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 554.154111ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................XXX.X...........................] duration:
432.214379ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:46:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 432.257859ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..........................X....X...X........X...X...........] duration:
438.580347ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:46:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 438.60827ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [.....................................XXXXXXX.XXXXXXXXXXX.XXX] duration:
1m0.023550467s trace:
https://jaeger.dev.storj.io/trace/52929149dff97569
FAIL (2025-12-10 23:46:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
]: Got next segment. Inline: false
Dec 10 23:45:43.515882472 (128h17m44.34359085s) [upload=133405]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:45:43.515888103 (128h17m44.343596494s) [upload=133405]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:45:43.541780231 (128h17m44.369488623s) [seg_pos={0 0}, upload=133405]: Begin upload segment...
Dec 10 23:45:43.541838031 (128h17m44.369546422s) [seg_pos={0 0}, upload=133405]: Done begin upload segment.
Dec 10 23:45:43.541895365 (128h17m44.369603759s) [noise=true , address=77.90.31.17:4792 , piece_id=VTNNV733IDRY7OTV7MVLSOZ3VDGILALWIUENJCZC5EDOG7LUGVHA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.542069948 (128h17m44.369778344s) [noise=true , address=145.79.227.28:4744 , piece_id=Z4QXM4ID5SKCK7RAIQKWUTLU3LLICMNBRZ3N4E7L5CRAZGMCXUKQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.542580661 (128h17m44.370289057s) [noise=true , address=64.49.13.169:4709 , piece_id=YOLRMERDQCKCDFCLBJVP5DFY64KVQOMXU7GFEBAROBFMMPKBEH4A, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.542783472 (128h17m44.370491872s) [noise=true , address=64.49.13.154:4933 , piece_id=PIEUGREHEDF6LRORMD27KSJNSNQGD6W36KOB7UTCWVCTRNLVIPVA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.542977500 (128h17m44.370685891s) [noise=true , address=145.79.227.27:4697 , piece_id=5OLE6PWJI5NOUKTHZPR6OYQ4TZ6DPQXB5MTY44OIQDFRNYJEE7VA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.543243387 (128h17m44.370951781s) [noise=true , address=77.90.31.19:4844 , piece_id=4YHTGEBKGVNZSFNTJKGXAQVSAQXU2KFWNJW4SJNKJSS2WX6KRM7Q, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.550603648 (128h17m44.378312051s) [noise=true , address=77.90.31.10:4524 , piece_id=WKK6VS4Q77HZFADV2SJSZAIPKUMBQBUUYAHBG7E5DSO55ANCUXNA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.554355047 (128h17m44.382063457s) [noise=true , address=45.91.116.8:4061 , piece_id=ZP7KZMATGIA7HCBTP7QVWL3YJJ64SWEHDUP6JTVOL5FQ73SIZGBQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.556376142 (128h17m44.384084542s) [noise=true , address=45.91.116.5:4309 , piece_id=ADPOS374QNDPECBIICRXQ3MYTJYIAM6DIL4H4A7XACEMVFBTLSTQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.556463808 (128h17m44.3841722s) [noise=true , address=45.91.116.6:4006 , piece_id=DGOQIGZVVHQU6OBCW44WC63DXGQU22B7N6LBF6EXNUB24LIIQBLQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.558980149 (128h17m44.386688543s) [noise=true , address=5.231.73.5:4007 , piece_id=SQBS36WNZLBSDALGKYR2FGQ354L2H3JB5COTA6GZ4SK72X4QUPNQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.560009670 (128h17m44.387718065s) [noise=true , address=45.91.116.11:4149 , piece_id=AZYP7IHLE76OHNN2VF5JRGRYGPTFUPKA7UIBOWBWEWO4EFUNOL6Q, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.561155295 (128h17m44.388863692s) [noise=true , address=145.79.227.25:4624 , piece_id=PTWIMLMKX5YCGWRUIAN2RIMS4LTF5RZNFIF7U5WRKLF246XBZ6JA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.561185862 (128h17m44.388894245s) [noise=true , address=77.90.31.21:4889 , piece_id=2QEO3WKB6E5TTSSVIMZRQBMZDJNKUYE2MT6FPXRW7WRLP7H3LZCQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.561828031 (128h17m44.389536423s) [noise=true , address=64.49.13.26:4953 , piece_id=27COZ6UHI2JZFXH57H5TFS7ECNVPB22MFFF2CEK5OG6FJY5FZ5SQ, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.562121307 (128h17m44.389829708s) [noise=true , address=145.79.227.26:4670 , piece_id=36FZSLW36JI3IEUWMHT6NWA5SUC6K7S6IV4T33R7RM3D6QZASGAA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.562349450 (128h17m44.390057842s) [noise=true , address=145.79.227.30:4790 , piece_id=RLUPGPLYBAHCP2JTUTZO27QS7R62NU5QIMKGLVR6FA7FGZPPNPEA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.563244970 (128h17m44.390953365s) [noise=true , address=77.90.31.14:4665 , piece_id=R2LZLZDYSXTFTB2CJW3UH5VN6WFJZND2TNOVAD32UUZ6HKIN7YXA, seg_pos={0 0}, upload=133405]: Uploading piece...
Dec 10 23:45:43.615186723 (128h17m44.442895113s) [upload=133405]: Next returned nil segment
Dec 10 23:45:43.615215824 (128h17m44.442924215s) [upload=133405]: Waiting for error group managing segments...
Dec 10 23:45:43.619707895 (128h17m44.447416287s) [noise=true , address=45.91.116.6:4006 , piece_id=DGOQIGZVVHQU6OBCW44WC63DXGQU22B7N6LBF6EXNUB24LIIQBLQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.620902097 (128h17m44.448610488s) [noise=true , address=45.91.116.8:4061 , piece_id=ZP7KZMATGIA7HCBTP7QVWL3YJJ64SWEHDUP6JTVOL5FQ73SIZGBQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.622669321 (128h17m44.450377722s) [noise=true , address=145.79.227.27:4697 , piece_id=5OLE6PWJI5NOUKTHZPR6OYQ4TZ6DPQXB5MTY44OIQDFRNYJEE7VA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.622907071 (128h17m44.450615462s) [noise=true , address=145.79.227.25:4624 , piece_id=PTWIMLMKX5YCGWRUIAN2RIMS4LTF5RZNFIF7U5WRKLF246XBZ6JA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.636202621 (128h17m44.463911012s) [noise=true , address=145.79.227.28:4744 , piece_id=Z4QXM4ID5SKCK7RAIQKWUTLU3LLICMNBRZ3N4E7L5CRAZGMCXUKQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.637143017 (128h17m44.464851409s) [noise=true , address=45.91.116.5:4309 , piece_id=ADPOS374QNDPECBIICRXQ3MYTJYIAM6DIL4H4A7XACEMVFBTLSTQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.643464782 (128h17m44.471173173s) [noise=true , address=145.79.227.26:4670 , piece_id=36FZSLW36JI3IEUWMHT6NWA5SUC6K7S6IV4T33R7RM3D6QZASGAA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.644008444 (128h17m44.471716842s) [noise=true , address=45.91.116.11:4149 , piece_id=AZYP7IHLE76OHNN2VF5JRGRYGPTFUPKA7UIBOWBWEWO4EFUNOL6Q, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:43.726966428 (128h17m44.55467482s) [noise=true , address=5.231.73.5:4007 , piece_id=SQBS36WNZLBSDALGKYR2FGQ354L2H3JB5COTA6GZ4SK72X4QUPNQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:44.174789034 (128h17m45.002497426s) [noise=true , address=64.49.13.26:4953 , piece_id=27COZ6UHI2JZFXH57H5TFS7ECNVPB22MFFF2CEK5OG6FJY5FZ5SQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:44.190577245 (128h17m45.018285651s) [noise=true , address=64.49.13.154:4933 , piece_id=PIEUGREHEDF6LRORMD27KSJNSNQGD6W36KOB7UTCWVCTRNLVIPVA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:44.603083641 (128h17m45.430792032s) [noise=true , address=145.79.227.30:4790 , piece_id=RLUPGPLYBAHCP2JTUTZO27QS7R62NU5QIMKGLVR6FA7FGZPPNPEA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:45:49.566037493 (128h17m50.393745891s) [noise=true , address=64.49.13.169:4709 , piece_id=YOLRMERDQCKCDFCLBJVP5DFY64KVQOMXU7GFEBAROBFMMPKBEH4A, seg_pos={0 0}, upload=133405]: Done uploading piece. err: <nil>
Dec 10 23:46:43.478798229 (128h18m44.306506632s) [noise=true , address=77.90.31.17:4792 , piece_id=VTNNV733IDRY7OTV7MVLSOZ3VDGILALWIUENJCZC5EDOG7LUGVHA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: ecclient: upload failed (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN, address:77.90.31.17:4792): protocol: rpc: dial tcp 77.90.31.17:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:43.479282317 (128h18m44.306990731s) [noise=true , address=77.90.31.14:4665 , piece_id=R2LZLZDYSXTFTB2CJW3UH5VN6WFJZND2TNOVAD32UUZ6HKIN7YXA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: ecclient: upload failed (node:12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, address:77.90.31.14:4665): protocol: rpc: dial tcp 77.90.31.14:4665: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:43.479604859 (128h18m44.30731326s) [noise=true , address=77.90.31.19:4844 , piece_id=4YHTGEBKGVNZSFNTJKGXAQVSAQXU2KFWNJW4SJNKJSS2WX6KRM7Q, seg_pos={0 0}, upload=133405]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 159.223.143.39:39826->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:43.479676076 (128h18m44.307384477s) [noise=true , address=77.90.31.21:4889 , piece_id=2QEO3WKB6E5TTSSVIMZRQBMZDJNKUYE2MT6FPXRW7WRLP7H3LZCQ, seg_pos={0 0}, upload=133405]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: write tcp 159.223.143.39:41700->77.90.31.21:4889: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:43.479891872 (128h18m44.307600273s) [noise=true , address=77.90.31.10:4524 , piece_id=WKK6VS4Q77HZFADV2SJSZAIPKUMBQBUUYAHBG7E5DSO55ANCUXNA, seg_pos={0 0}, upload=133405]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: write tcp 159.223.143.39:41930->77.90.31.10:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:43.479928107 (128h18m44.307636499s) [seg_pos={0 0}, upload=133405]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 10 23:46:43.499575864 (128h18m44.327284255s) [upload=133405]: Done uploading segments...
Dec 10 23:46:43.499623200 (128h18m44.327331596s) [upload=133405]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.023566963s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................XXX.............................] duration:
665.632719ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:46:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 665.652688ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...........................X....X...X........X...X..........] duration:
457.565372ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:46:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 457.593101ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
19 out of 60 is failed (threshold=3) [.....................................XX..XXXXXX.XXXXXX.XXXXX] duration:
1m0.027952561s trace:
https://jaeger.dev.storj.io/trace/2c61f37ea9f4b947
FAIL (2025-12-10 23:46:24 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:45:24.212429606 (128h17m25.040137998s) [upload=133398]: Starting upload
Dec 10 23:45:24.212478837 (128h17m25.040187237s) [upload=133398]: Done starting upload
Dec 10 23:45:24.212828766 (128h17m25.040537177s) [upload=133398]: Uploading segments...
Dec 10 23:45:24.212990454 (128h17m25.040698855s) [upload=133398]: Got next segment. Inline: false
Dec 10 23:45:24.213074330 (128h17m25.04078273s) [upload=133398]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:45:24.213081749 (128h17m25.040790149s) [upload=133398]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:45:24.242415476 (128h17m25.070123872s) [seg_pos={0 0}, upload=133398]: Begin upload segment...
Dec 10 23:45:24.242502004 (128h17m25.070210396s) [seg_pos={0 0}, upload=133398]: Done begin upload segment.
Dec 10 23:45:24.242524836 (128h17m25.070233238s) [noise=true , address=77.90.31.19:4850 , piece_id=UH3E2CGCM2FLCOJT32DBHMWZRZHIO35GEWHFRM5PRSRDD4QZQVOA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.242611004 (128h17m25.070319397s) [noise=true , address=77.90.31.18:4839 , piece_id=WUGXTLG4TN6WTF3AOXXTPIQ75M2PXZBNACXPVWQIJ6H3UD2W6BJQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.243178872 (128h17m25.070887273s) [noise=true , address=145.79.227.31:4808 , piece_id=4QXI2DOQUJFDFEPBIUJ7M627ZUHL2PTIRN4QBPZCYCUP4LSWGDGQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.243864282 (128h17m25.07157268s) [noise=true , address=45.91.116.4:4959 , piece_id=4FVEYL7VN5FWDNCSCLEZJY5HSX6CB6NE5O55JPF5F55V76HOJXFA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.245180811 (128h17m25.072889207s) [noise=true , address=45.91.116.13:4166 , piece_id=YSH5KC6KWHKDPGLQBYJOAUR352JWASGFUHTEMQ2QYPSGVZVVH5AA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.245437219 (128h17m25.073145614s) [noise=true , address=45.91.116.12:4243 , piece_id=OR4ZKWVHXPS4JGOMOGKPPDYXY5JLHSWS37B7ZMVAQNOMGMGHHTRA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.245969749 (128h17m25.073678144s) [noise=true , address=145.79.227.28:4746 , piece_id=NRNXDP56ORSEG24VNWDUKTTTOX2ZO6OYJKZBYHXZAY2EZ25KITEQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.246258047 (128h17m25.073966428s) [noise=true , address=145.79.227.30:4776 , piece_id=A52EIAUK5UQRJTHCT3FEQGQVBJ7TIDNB4YVZ5OTTS7IBTCEZADSQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.246711138 (128h17m25.07441954s) [noise=true , address=64.49.13.169:4711 , piece_id=2NR3VFMHJQCU7QW3YG4U2CSWIBEX744LATX542XUJAONG6U4JAUA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.246719307 (128h17m25.074427684s) [noise=true , address=5.231.73.2:4021 , piece_id=656KW4YIIZENIB3QOLNNZL5SFZ6EI3MUR4O6USW2WMHIZGDSFPOA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.247343727 (128h17m25.075052132s) [noise=true , address=77.90.31.14:4668 , piece_id=DZDUR7XJTJBBTJNKFN6NTLLA4FIOAGWRKMYHHIDJLKSTBL4BNWGQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.247922387 (128h17m25.07563079s) [noise=true , address=77.90.31.11:4582 , piece_id=OCEQ45CMCD2D4EE7KL2RWKTEYA4JZ7BKPAA5E7H7PKKWNNACT2XQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.248425582 (128h17m25.076133984s) [noise=true , address=77.90.31.17:4803 , piece_id=56CB6SEPDZ4PWMFPTOLDU7AOTYTSRKYB2B2JHUGXDAPZVNEUOIEA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.249105289 (128h17m25.076813686s) [noise=true , address=77.90.31.15:4712 , piece_id=GRN3GYQUCQQKLKR4IKQOIVUPP5H5GI27EQWYULHN4UKXJPGJEXNQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.249758441 (128h17m25.077466844s) [noise=true , address=145.79.227.44:4218 , piece_id=UOFLKAOWAWN3J5BU2F3B534D7ABYE2YXUCDYG7HVZMAY25KYCHMA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.250322512 (128h17m25.078030907s) [noise=true , address=45.91.116.6:4990 , piece_id=OU3YNIFOP4GALLHVCHVGPD65QB6YOUE7NYB5XHTXH3XCDZDGRBVA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.250827537 (128h17m25.078535932s) [noise=true , address=145.79.227.25:4624 , piece_id=WDR3EL6A7UUKZVRPMMNIVV3Q72F5FJAWMA3ROJHLCQFSE44HP7DQ, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.251363842 (128h17m25.079072234s) [noise=true , address=145.79.227.21:4494 , piece_id=24MNF46RV7EDSP4WWAULKCWVWBNXYK7Q3OB5N67NJDRSJ4ZE6CMA, seg_pos={0 0}, upload=133398]: Uploading piece...
Dec 10 23:45:24.316757825 (128h17m25.144466217s) [upload=133398]: Next returned nil segment
Dec 10 23:45:24.316775245 (128h17m25.144483637s) [upload=133398]: Waiting for error group managing segments...
Dec 10 23:45:24.320125462 (128h17m25.147833853s) [noise=true , address=45.91.116.13:4166 , piece_id=YSH5KC6KWHKDPGLQBYJOAUR352JWASGFUHTEMQ2QYPSGVZVVH5AA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.320194441 (128h17m25.147902827s) [noise=true , address=45.91.116.12:4243 , piece_id=OR4ZKWVHXPS4JGOMOGKPPDYXY5JLHSWS37B7ZMVAQNOMGMGHHTRA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.320969102 (128h17m25.148677494s) [noise=true , address=45.91.116.4:4959 , piece_id=4FVEYL7VN5FWDNCSCLEZJY5HSX6CB6NE5O55JPF5F55V76HOJXFA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.321457569 (128h17m25.149165965s) [noise=true , address=45.91.116.6:4990 , piece_id=OU3YNIFOP4GALLHVCHVGPD65QB6YOUE7NYB5XHTXH3XCDZDGRBVA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.322717286 (128h17m25.150425685s) [noise=true , address=145.79.227.21:4494 , piece_id=24MNF46RV7EDSP4WWAULKCWVWBNXYK7Q3OB5N67NJDRSJ4ZE6CMA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.324733321 (128h17m25.152441721s) [noise=true , address=145.79.227.28:4746 , piece_id=NRNXDP56ORSEG24VNWDUKTTTOX2ZO6OYJKZBYHXZAY2EZ25KITEQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.359687052 (128h17m25.187395442s) [noise=true , address=145.79.227.25:4624 , piece_id=WDR3EL6A7UUKZVRPMMNIVV3Q72F5FJAWMA3ROJHLCQFSE44HP7DQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.364292248 (128h17m25.192000639s) [noise=true , address=145.79.227.31:4808 , piece_id=4QXI2DOQUJFDFEPBIUJ7M627ZUHL2PTIRN4QBPZCYCUP4LSWGDGQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:24.955671497 (128h17m25.783379896s) [noise=true , address=5.231.73.2:4021 , piece_id=656KW4YIIZENIB3QOLNNZL5SFZ6EI3MUR4O6USW2WMHIZGDSFPOA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:25.262226737 (128h17m26.089935128s) [noise=true , address=145.79.227.44:4218 , piece_id=UOFLKAOWAWN3J5BU2F3B534D7ABYE2YXUCDYG7HVZMAY25KYCHMA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:25.303826833 (128h17m26.131535252s) [noise=true , address=145.79.227.30:4776 , piece_id=A52EIAUK5UQRJTHCT3FEQGQVBJ7TIDNB4YVZ5OTTS7IBTCEZADSQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:45:30.320974126 (128h17m31.148682521s) [noise=true , address=64.49.13.169:4711 , piece_id=2NR3VFMHJQCU7QW3YG4U2CSWIBEX744LATX542XUJAONG6U4JAUA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: <nil>
Dec 10 23:46:24.178196253 (128h18m25.005904654s) [noise=true , address=77.90.31.14:4668 , piece_id=DZDUR7XJTJBBTJNKFN6NTLLA4FIOAGWRKMYHHIDJLKSTBL4BNWGQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: ecclient: upload failed (node:1U4SKog2kLXNyjyWs7Jg1MGerbhSYLwJUbskE3TGCHJpNWQHS7, address:77.90.31.14:4668): protocol: rpc: dial tcp 77.90.31.14:4668: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:24.178243229 (128h18m25.005951643s) [noise=true , address=77.90.31.15:4712 , piece_id=GRN3GYQUCQQKLKR4IKQOIVUPP5H5GI27EQWYULHN4UKXJPGJEXNQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: write tcp 159.223.143.39:48108->77.90.31.15:4712: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:24.178811146 (128h18m25.006519538s) [noise=true , address=77.90.31.17:4803 , piece_id=56CB6SEPDZ4PWMFPTOLDU7AOTYTSRKYB2B2JHUGXDAPZVNEUOIEA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: ecclient: upload failed (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y, address:77.90.31.17:4803): protocol: rpc: dial tcp 77.90.31.17:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:24.179061784 (128h18m25.006770189s) [noise=true , address=77.90.31.11:4582 , piece_id=OCEQ45CMCD2D4EE7KL2RWKTEYA4JZ7BKPAA5E7H7PKKWNNACT2XQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:24.179234470 (128h18m25.006942872s) [noise=true , address=77.90.31.19:4850 , piece_id=UH3E2CGCM2FLCOJT32DBHMWZRZHIO35GEWHFRM5PRSRDD4QZQVOA, seg_pos={0 0}, upload=133398]: Done uploading piece. err: ecclient: upload failed (node:12oGsNdVAKAabvVuWFZ2mpydVYjzhJycig4cqdcE5oAS2YsHmYv, address:77.90.31.19:4850): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:46:24.179364646 (128h18m25.007073051s) [noise=true , address=77.90.31.18:4839 , piece_id=WUGXTLG4TN6WTF3AOXXTPIQ75M2PXZBNACXPVWQIJ6H3UD2W6BJQ, seg_pos={0 0}, upload=133398]: Done uploading piece. err: ecclient: upload failed (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx, address:77.90.31.18:4839): protocol: write tcp 159.223.143.39:46736->77.90.31.18:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:46:24.179401322 (128h18m25.007109718s) [seg_pos={0 0}, upload=133398]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:46:24.202169082 (128h18m25.029877474s) [upload=133398]: Done uploading segments...
Dec 10 23:46:24.202247790 (128h18m25.029956183s) [upload=133398]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.028033973s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................XXX............................] duration:
523.067235ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:46:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 523.090452ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................XXX.X..........................] duration:
419.827531ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:46:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 419.86693ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration:
541.194352ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:45:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 541.222585ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............................XXX.X.........................] duration:
440.760561ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:45:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 440.787685ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
20 out of 60 is failed (threshold=3) [......................................XXXXXXX.XXXXXXXXXXX.XX] duration:
1m0.02746605s trace:
https://jaeger.dev.storj.io/trace/6e91018c3f277116
FAIL (2025-12-10 23:45:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
ue , address=145.79.227.31:4826 , piece_id=WKIXNI6FMUSLIXLYQ666D4LNN72WH66AT6PZV3HXCXGDC2SGQDYA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.446992022 (128h16m44.274700424s) [noise=true , address=77.90.31.15:4724 , piece_id=26YX6ISAXHBZ24VKPKRDSGEZFFGRG44E47A57LSENZAG2UTVQSPQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.447160532 (128h16m44.274868934s) [noise=true , address=77.90.31.19:4855 , piece_id=P7PGZO63S5JIOIAWGZCMFMC4AKR64XDJHWS6J2XFZJCRNJRTSENA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.447723758 (128h16m44.275432156s) [noise=true , address=77.90.31.12:4600 , piece_id=HSA5MMC32V6A2DDGTEWSGL7L5ANEA4NWXDDZOL4LK2AAVLIZ5RCQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.448671552 (128h16m44.276379957s) [noise=true , address=77.90.31.15:4701 , piece_id=YDW4N6BCGF4NY5LL75PXAVBBPMH275X2QJI5KJHLZGEDD627WIJA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.448905233 (128h16m44.276613635s) [noise=true , address=77.90.31.16:4762 , piece_id=NS62RYSM6GHXLGRI522TCYCNM4X4OFZDSIIDXM6LUTGW3HIYFJUQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.449871440 (128h16m44.277579836s) [noise=true , address=145.79.227.21:4500 , piece_id=VYANNQC3F3J5LATDYZP4J2CGQOH27BTSQRWNQJFQL4SFNORLYNUA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.450375446 (128h16m44.278083838s) [noise=true , address=77.90.31.13:4632 , piece_id=2ATFDOXINQALRMSOTYKWBN6ND6VUMMS3JSGLUE6FBWGH4C73QCSA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.450896733 (128h16m44.278605135s) [noise=true , address=145.79.227.27:4703 , piece_id=FENZMXEDDCSTDUHVZFEFNKKTR24CANPE6HG4AQE2V2UVC2IVHHBA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.451126236 (128h16m44.278834658s) [noise=true , address=5.231.73.5:4029 , piece_id=HLCOIU4ZFZMQAL7HG2DBOCTWTGNYF2ZE7R5SVVS6TINPCNUXDRAA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.451506826 (128h16m44.27921523s) [noise=true , address=77.90.31.14:4680 , piece_id=VKR6EEF2WHYM2AC6PKPZEW3LMNZMQOIFQUW2BYK563I3JUI4VLPA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.451589461 (128h16m44.279297864s) [noise=true , address=45.91.116.10:4112 , piece_id=6JLOTORRFRQYNBGY37GFLZCMMLZCBU4UYMOGW7DQ3OGC6E2XQICA, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.451999280 (128h16m44.279707682s) [noise=true , address=145.79.227.24:4592 , piece_id=UUWWD7XKXD7TBACEZOBEQ4OVPKCIPCZM2APYAAWVOLL3WNORKUVQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.452057253 (128h16m44.279765646s) [noise=true , address=145.79.227.44:4214 , piece_id=DYOD2YAFKET2AQX7MLYLBTCOFHDWFE4ULDQ2ZQVVO5JDY6LUTLCQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.452411112 (128h16m44.280119503s) [noise=true , address=77.90.31.10:4541 , piece_id=NPTF26EWOJTGI5WTSTEX75XAY7JKPS2TCFF3IQWCOSOHOGVBK7EQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.452939003 (128h16m44.280647403s) [noise=true , address=45.91.116.14:4211 , piece_id=X7WCPXG5TPWKZQAEJ6A42RSZ7BAPYZLX2RWAZ3S2LMTUQWJHIMOQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.455579103 (128h16m44.2832875s) [noise=true , address=145.79.227.33:4899 , piece_id=3WQCUEBBD3OXOHSH4TDMJ6UXZTQRB4ZUHMLXB2EJQP3MN6PMMYYQ, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.456252172 (128h16m44.283960569s) [noise=true , address=45.91.116.9:4276 , piece_id=QOOS2LKTATRQRMPBNMSW6MVBD5E4KVATU52YNF3U77NAIAIRFP6Q, seg_pos={0 0}, upload=133386]: Uploading piece...
Dec 10 23:44:43.529267904 (128h16m44.356976298s) [upload=133386]: Next returned nil segment
Dec 10 23:44:43.529294138 (128h16m44.357002541s) [upload=133386]: Waiting for error group managing segments...
Dec 10 23:44:43.537122588 (128h16m44.36483098s) [noise=true , address=145.79.227.27:4703 , piece_id=FENZMXEDDCSTDUHVZFEFNKKTR24CANPE6HG4AQE2V2UVC2IVHHBA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.537540497 (128h16m44.365248888s) [noise=true , address=145.79.227.21:4500 , piece_id=VYANNQC3F3J5LATDYZP4J2CGQOH27BTSQRWNQJFQL4SFNORLYNUA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.541673957 (128h16m44.36938235s) [noise=true , address=145.79.227.24:4592 , piece_id=UUWWD7XKXD7TBACEZOBEQ4OVPKCIPCZM2APYAAWVOLL3WNORKUVQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.541788012 (128h16m44.369496405s) [noise=true , address=145.79.227.33:4899 , piece_id=3WQCUEBBD3OXOHSH4TDMJ6UXZTQRB4ZUHMLXB2EJQP3MN6PMMYYQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.548310922 (128h16m44.376019323s) [noise=true , address=45.91.116.14:4211 , piece_id=X7WCPXG5TPWKZQAEJ6A42RSZ7BAPYZLX2RWAZ3S2LMTUQWJHIMOQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.549535332 (128h16m44.377243727s) [noise=true , address=45.91.116.10:4112 , piece_id=6JLOTORRFRQYNBGY37GFLZCMMLZCBU4UYMOGW7DQ3OGC6E2XQICA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.550348352 (128h16m44.378056744s) [noise=true , address=45.91.116.9:4276 , piece_id=QOOS2LKTATRQRMPBNMSW6MVBD5E4KVATU52YNF3U77NAIAIRFP6Q, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.553820052 (128h16m44.381528429s) [noise=true , address=145.79.227.31:4826 , piece_id=WKIXNI6FMUSLIXLYQ666D4LNN72WH66AT6PZV3HXCXGDC2SGQDYA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:43.634018149 (128h16m44.461726547s) [noise=true , address=5.231.73.5:4029 , piece_id=HLCOIU4ZFZMQAL7HG2DBOCTWTGNYF2ZE7R5SVVS6TINPCNUXDRAA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:44:44.498491369 (128h16m45.326199751s) [noise=true , address=145.79.227.44:4214 , piece_id=DYOD2YAFKET2AQX7MLYLBTCOFHDWFE4ULDQ2ZQVVO5JDY6LUTLCQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: <nil>
Dec 10 23:45:43.389172946 (128h17m44.216881347s) [noise=true , address=77.90.31.13:4632 , piece_id=2ATFDOXINQALRMSOTYKWBN6ND6VUMMS3JSGLUE6FBWGH4C73QCSA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:43.389362260 (128h17m44.217070657s) [noise=true , address=77.90.31.15:4701 , piece_id=YDW4N6BCGF4NY5LL75PXAVBBPMH275X2QJI5KJHLZGEDD627WIJA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:12u5UY7VzJcjCvHtATf2ofXugWew2JXVXBjpManmX4wSzQwR7A6, address:77.90.31.15:4701): protocol: write tcp 159.223.143.39:51656->77.90.31.15:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:43.389482234 (128h17m44.217190634s) [noise=true , address=77.90.31.19:4855 , piece_id=P7PGZO63S5JIOIAWGZCMFMC4AKR64XDJHWS6J2XFZJCRNJRTSENA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: rpc: dial tcp 77.90.31.19:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:43.389590803 (128h17m44.217299203s) [noise=true , address=77.90.31.10:4541 , piece_id=NPTF26EWOJTGI5WTSTEX75XAY7JKPS2TCFF3IQWCOSOHOGVBK7EQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: rpc: dial tcp 77.90.31.10:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:43.389808002 (128h17m44.217516402s) [noise=true , address=77.90.31.14:4680 , piece_id=VKR6EEF2WHYM2AC6PKPZEW3LMNZMQOIFQUW2BYK563I3JUI4VLPA, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: rpc: dial tcp 77.90.31.14:4680: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 10 23:45:43.389833365 (128h17m44.217541749s) [noise=true , address=77.90.31.12:4600 , piece_id=HSA5MMC32V6A2DDGTEWSGL7L5ANEA4NWXDDZOL4LK2AAVLIZ5RCQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:43.389862279 (128h17m44.217570679s) [noise=true , address=77.90.31.16:4762 , piece_id=NS62RYSM6GHXLGRI522TCYCNM4X4OFZDSIIDXM6LUTGW3HIYFJUQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:45:43.390075133 (128h17m44.217783531s) [noise=true , address=77.90.31.15:4724 , piece_id=26YX6ISAXHBZ24VKPKRDSGEZFFGRG44E47A57LSENZAG2UTVQSPQ, seg_pos={0 0}, upload=133386]: Done uploading piece. err: ecclient: upload failed (node:12YEcohZBsQSm45orPuT8np4EGgXQaWAM8rCsmUfA1PfNubwLmo, address:77.90.31.15:4724): protocol: rpc: dial tcp 77.90.31.15:4724: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:43.390113400 (128h17m44.217821784s) [seg_pos={0 0}, upload=133386]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:45:43.413821838 (128h17m44.24153023s) [upload=133386]: Done uploading segments...
Dec 10 23:45:43.413857844 (128h17m44.241566239s) [upload=133386]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.027494197s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [............................X....X...X........X...X.........] duration:
474.981144ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:45:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 475.007751ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration:
586.520196ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:45:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 586.552914ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.............................X....X...X........X...X........] duration:
437.67007ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:45:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 437.697048ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................XXX.X........................] duration:
427.991121ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:45:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 428.016504ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
18 out of 60 is failed (threshold=3) [......................................XX..XXXXXX.XXXXXX.XXXX] duration:
1m0.024653228s trace:
https://jaeger.dev.storj.io/trace/20247941c66ae2b3
FAIL (2025-12-10 23:45:12 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
Dec 10 23:44:12.580035287 (128h16m13.407743684s) [upload=133376]: Starting upload
Dec 10 23:44:12.580080848 (128h16m13.407789225s) [upload=133376]: Done starting upload
Dec 10 23:44:12.580117198 (128h16m13.407825579s) [upload=133376]: Uploading segments...
Dec 10 23:44:12.580330037 (128h16m13.408038415s) [upload=133376]: Got next segment. Inline: false
Dec 10 23:44:12.580373883 (128h16m13.408082519s) [upload=133376]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:44:12.580381948 (128h16m13.408090339s) [upload=133376]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:44:12.608797371 (128h16m13.436505778s) [seg_pos={0 0}, upload=133376]: Begin upload segment...
Dec 10 23:44:12.608879857 (128h16m13.436588248s) [seg_pos={0 0}, upload=133376]: Done begin upload segment.
Dec 10 23:44:12.608908391 (128h16m13.436616793s) [noise=true , address=64.49.13.140:4511 , piece_id=F5MQDRUOLHM4ZHDXD65XYKVIPO7CBC5BPZMGWMY45H6TBE7HRR2A, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.608931816 (128h16m13.436640217s) [noise=true , address=77.90.31.15:4723 , piece_id=UALWWPH6ZHACVTKMU5RHBMWJ655VOYKZBICAVPIR2W7LACLKAFDA, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.608986971 (128h16m13.436695366s) [noise=true , address=5.231.73.3:4005 , piece_id=76RYBKIVKNLHG35XF2MTSOER5UTAUS6WDBWIS2VNMGONQTINIPOA, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.609493410 (128h16m13.437201813s) [noise=true , address=77.90.31.16:4740 , piece_id=TTXRQCB5WIX6CJ6WLDDTCPY24F2QUCS5R4B3QSCJHNOR4TSCKG5A, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.610041373 (128h16m13.437749755s) [noise=true , address=145.79.227.16:4935 , piece_id=ZYXDY2N4UYUB3XGIVRQJIPG527AVVO2G7H56V7ENDW7VSO73FZYA, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.611239453 (128h16m13.438947846s) [noise=true , address=77.90.31.19:4870 , piece_id=CJML4QRX5ZPRIOXQCC3KUOGN3MERDCI5MXPECN27IJ3H6AFPQMKQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.611660037 (128h16m13.439368432s) [noise=true , address=145.79.227.26:4657 , piece_id=F7KYXMBT5EO2MV5VD7UUAKTZ7EEHGLFEF3JQTHZRGXB2BYO4M5AQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.612023594 (128h16m13.439731988s) [noise=true , address=45.91.116.5:4327 , piece_id=AIBG22QIVJD3C3K5IN4ZHTWYXE6QL7IDRWCHQCJIA5UO5ODDCV4A, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.612437729 (128h16m13.440146126s) [noise=true , address=145.79.227.31:4806 , piece_id=SMPPCAY3PIFHQMULYS7CY5FZQX7MEMUMKNACFNOUTLCU4BVU5F7Q, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.612917083 (128h16m13.440625477s) [noise=true , address=77.90.31.21:4884 , piece_id=XCDSRGTT4JC2SRQZBTOE3BZGVLZBXDSRJ52CJDX3O2WDGB3KUMIQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.613305000 (128h16m13.441013394s) [noise=true , address=145.79.227.26:4682 , piece_id=LUKPC5XLVKHAY6P6W5TYT3VJA4P7LPGKKT52JIBGHBF4SGI733PQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.613780877 (128h16m13.441489274s) [noise=true , address=45.91.116.6:4979 , piece_id=4YEGM7RJR6QIEMNOD4ZVWBHKQATIJTA6G32GNKJAPIQPSOBZH4YA, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.614261732 (128h16m13.441970127s) [noise=true , address=145.79.227.34:4927 , piece_id=2X7OAOV4UYZPFYUZXGM4OGSCAJDI5OY5TYWGZQFMOHGRY2CTVEZQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.614759117 (128h16m13.442467512s) [noise=true , address=77.90.31.12:4619 , piece_id=FA6T76GJMR6ANDT7A4U7YDYIFBOPD62M3USIBLDVX76ONE2HZ2EQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.616394033 (128h16m13.444102433s) [noise=true , address=45.91.116.13:4168 , piece_id=MJUXC47X6R2QRID2CAEJHQZOQXUHGBVF7JZSABW3OTSGUDAGUIYA, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.617904625 (128h16m13.445613019s) [noise=true , address=45.91.116.12:4235 , piece_id=RD4FFPNQLYVY35BFRCP7755IPSMRTBNSNOLHSCWXKYL2T47HTVNQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.619388134 (128h16m13.447096538s) [noise=true , address=45.91.116.14:4220 , piece_id=57ZEGX7RY73ZBU55F5ONEAEMPPDZZXZRE5CJD2QRXKRCNBX2HIXQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.620932965 (128h16m13.448641365s) [noise=true , address=77.90.31.10:4522 , piece_id=6KB6MFEPVCIBSBAKAFLAGNCDUBKZEXZKG2CYGFIPW255CGQDCPIQ, seg_pos={0 0}, upload=133376]: Uploading piece...
Dec 10 23:44:12.694731245 (128h16m13.522439627s) [upload=133376]: Next returned nil segment
Dec 10 23:44:12.694744806 (128h16m13.522453198s) [upload=133376]: Waiting for error group managing segments...
Dec 10 23:44:12.699629075 (128h16m13.52733747s) [noise=true , address=45.91.116.6:4979 , piece_id=4YEGM7RJR6QIEMNOD4ZVWBHKQATIJTA6G32GNKJAPIQPSOBZH4YA, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.701172557 (128h16m13.528880952s) [noise=true , address=145.79.227.26:4682 , piece_id=LUKPC5XLVKHAY6P6W5TYT3VJA4P7LPGKKT52JIBGHBF4SGI733PQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.701595027 (128h16m13.529303418s) [noise=true , address=145.79.227.26:4657 , piece_id=F7KYXMBT5EO2MV5VD7UUAKTZ7EEHGLFEF3JQTHZRGXB2BYO4M5AQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.703819988 (128h16m13.531528389s) [noise=true , address=145.79.227.34:4927 , piece_id=2X7OAOV4UYZPFYUZXGM4OGSCAJDI5OY5TYWGZQFMOHGRY2CTVEZQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.714013498 (128h16m13.541721889s) [noise=true , address=45.91.116.12:4235 , piece_id=RD4FFPNQLYVY35BFRCP7755IPSMRTBNSNOLHSCWXKYL2T47HTVNQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.717220794 (128h16m13.544929185s) [noise=true , address=45.91.116.5:4327 , piece_id=AIBG22QIVJD3C3K5IN4ZHTWYXE6QL7IDRWCHQCJIA5UO5ODDCV4A, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.717750261 (128h16m13.545458639s) [noise=true , address=45.91.116.14:4220 , piece_id=57ZEGX7RY73ZBU55F5ONEAEMPPDZZXZRE5CJD2QRXKRCNBX2HIXQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.718154654 (128h16m13.545863045s) [noise=true , address=45.91.116.13:4168 , piece_id=MJUXC47X6R2QRID2CAEJHQZOQXUHGBVF7JZSABW3OTSGUDAGUIYA, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.718866388 (128h16m13.546574779s) [noise=true , address=145.79.227.31:4806 , piece_id=SMPPCAY3PIFHQMULYS7CY5FZQX7MEMUMKNACFNOUTLCU4BVU5F7Q, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:12.789067215 (128h16m13.616775607s) [noise=true , address=5.231.73.3:4005 , piece_id=76RYBKIVKNLHG35XF2MTSOER5UTAUS6WDBWIS2VNMGONQTINIPOA, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:13.273322436 (128h16m14.101030828s) [noise=true , address=64.49.13.140:4511 , piece_id=F5MQDRUOLHM4ZHDXD65XYKVIPO7CBC5BPZMGWMY45H6TBE7HRR2A, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:44:13.771487884 (128h16m14.599196276s) [noise=true , address=145.79.227.16:4935 , piece_id=ZYXDY2N4UYUB3XGIVRQJIPG527AVVO2G7H56V7ENDW7VSO73FZYA, seg_pos={0 0}, upload=133376]: Done uploading piece. err: <nil>
Dec 10 23:45:12.548768842 (128h17m13.376477221s) [noise=true , address=77.90.31.21:4884 , piece_id=XCDSRGTT4JC2SRQZBTOE3BZGVLZBXDSRJ52CJDX3O2WDGB3KUMIQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: ecclient: upload failed (node:12aqgkDxFY3xZa9s2vq1DdCFFX2725aDicrxfvoYuXPfe4ozkHV, address:77.90.31.21:4884): protocol: write tcp 159.223.143.39:59748->77.90.31.21:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:12.548956413 (128h17m13.376664805s) [noise=true , address=77.90.31.12:4619 , piece_id=FA6T76GJMR6ANDT7A4U7YDYIFBOPD62M3USIBLDVX76ONE2HZ2EQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: write tcp 159.223.143.39:53714->77.90.31.12:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:12.549090152 (128h17m13.37679853s) [noise=true , address=77.90.31.10:4522 , piece_id=6KB6MFEPVCIBSBAKAFLAGNCDUBKZEXZKG2CYGFIPW255CGQDCPIQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: ecclient: upload failed (node:12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, address:77.90.31.10:4522): protocol: write tcp 159.223.143.39:51192->77.90.31.10:4522: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:12.549216584 (128h17m13.376924962s) [noise=true , address=77.90.31.16:4740 , piece_id=TTXRQCB5WIX6CJ6WLDDTCPY24F2QUCS5R4B3QSCJHNOR4TSCKG5A, seg_pos={0 0}, upload=133376]: Done uploading piece. err: ecclient: upload failed (node:145raYQJozNw3SMbj78wDsR6EjPvJTqqcocf1SaPs3HEnNVS5p, address:77.90.31.16:4740): protocol: write tcp 159.223.143.39:43350->77.90.31.16:4740: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:12.549323810 (128h17m13.377032188s) [noise=true , address=77.90.31.15:4723 , piece_id=UALWWPH6ZHACVTKMU5RHBMWJ655VOYKZBICAVPIR2W7LACLKAFDA, seg_pos={0 0}, upload=133376]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: write tcp 159.223.143.39:37010->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:12.549461306 (128h17m13.3771697s) [noise=true , address=77.90.31.19:4870 , piece_id=CJML4QRX5ZPRIOXQCC3KUOGN3MERDCI5MXPECN27IJ3H6AFPQMKQ, seg_pos={0 0}, upload=133376]: Done uploading piece. err: ecclient: upload failed (node:1hNVXEVe8DW8F8meERGWkxAazEZKLKBeprNn6e5C94eEoAMBaR, address:77.90.31.19:4870): protocol: write tcp 159.223.143.39:47498->77.90.31.19:4870: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:45:12.549495898 (128h17m13.377204276s) [seg_pos={0 0}, upload=133376]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:45:12.570603134 (128h17m13.398311526s) [upload=133376]: Done uploading segments...
Dec 10 23:45:12.570636811 (128h17m13.398345201s) [upload=133376]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.024667499s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..............................X....X...X........X...X.......] duration:
499.952206ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:45:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 499.987048ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [.......................................XXXXXXX.XXXXXXXXXXX.X] duration:
1m0.031912427s trace:
https://jaeger.dev.storj.io/trace/43650d40e93050f5
FAIL (2025-12-10 23:44:43 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
JPVBNGHPQCJDIBZMMUZ6YNWQZHBOGU5QVHZ2Q6A, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.378512470 (128h15m44.206220851s) [noise=true , address=77.90.31.15:4712 , piece_id=TCWKRXUJJUQYGXVQDET4WUU5IMPCZ74A43KCJ2T3A5K6H57IQ4AA, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.379061534 (128h15m44.206769928s) [noise=true , address=45.91.116.10:4092 , piece_id=O2HDXSWQVNKGHZ7W5AWVPCGG3TJ4S4V7RLKT5XYGVHUBLYJHIZQQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.379589947 (128h15m44.207298345s) [noise=true , address=77.90.31.14:4682 , piece_id=IMMATXQ23SDDXWSLXM73IIV7EAV4ZYIS23FLBYXQRDCNDTUCILQQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.379611106 (128h15m44.207319503s) [noise=true , address=145.79.227.39:4047 , piece_id=6IPUBEPRB62GKJ3PZIFYPWC54TZVNPTZFYORNFGZWUIXPYC4EZQQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.380066047 (128h15m44.207774445s) [noise=true , address=145.79.227.12:4798 , piece_id=4KRVETRCIKRCWA6CADS3YW5XFZM25IWYEOGCGXT563LAEJZWE7LQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.380100966 (128h15m44.207809356s) [noise=true , address=77.90.31.13:4653 , piece_id=AZSQXGFVP3WDU5URMNZLVAR3WIZWTDIKBNBYGD7PIYZP72CJK22A, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.380429822 (128h15m44.208138214s) [noise=true , address=77.90.31.16:4759 , piece_id=ID3XX6VXIRB5GAQHSQ6KEG6KYLPLJGSQULKUUUAGM6WBCNP7MVHA, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.380682389 (128h15m44.208390793s) [noise=true , address=45.91.116.13:4179 , piece_id=LELMW36GJQYKM7HQJSF2ODQGLAT7R67AMLG4MQPVUGOFSH6QUAGQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.380871228 (128h15m44.20857962s) [noise=true , address=45.91.116.11:4133 , piece_id=I7J5DKPUGNGKQLDM74CYKNZXQUM7M3QQ4VCWMKPAFHPI6M756SNQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.381150889 (128h15m44.208859291s) [noise=true , address=77.90.31.10:4534 , piece_id=33CMFWHSKYSJUD4BX3CAT3FCSDD6YLGEALYJ3RFYLRDBE5X2SPRA, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.381240067 (128h15m44.208948461s) [noise=true , address=45.91.116.9:4270 , piece_id=BTWWCBF64DXUORXC24CD2KG4VAMREXHZ4RSGCXAROLWGFO5VBQ6A, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.381624210 (128h15m44.209332605s) [noise=true , address=77.90.31.19:4860 , piece_id=Q2U64XB7AK4IT35AR6OPOYS6LGYGUUGALYRO522URUUZRC6IGLUQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.381666668 (128h15m44.20937506s) [noise=true , address=145.79.227.30:4794 , piece_id=7OLDNEYASADLBCX6EOIWLDZ24X366XGSQKVEUNX2TNZCH7KZDWBA, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.382007489 (128h15m44.209715881s) [noise=true , address=145.79.227.34:4937 , piece_id=3PX4TT5HLZW77T7SRYWWQGD24DCPPLLS5QRGG6DNSD7AYVSXDQLA, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.382051644 (128h15m44.209760035s) [noise=true , address=145.79.227.52:4466 , piece_id=AHS6KJVURYU5REAHEPTYQB4AKHJVDQUGLEFMLJV4M6RXLR46PMZQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.383760775 (128h15m44.211469168s) [noise=true , address=77.90.31.21:4910 , piece_id=5LDSTTLUR7V7WABWWMS6OD6ZWHRCVAN3W4BOOFTAZAABVN5V573Q, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.384674867 (128h15m44.21238325s) [noise=true , address=145.79.227.2:4137 , piece_id=MD6LEPAJPQW5G4TWRHRKVPPZPCETLRPREVLLB5AYGU5PZANUOGDQ, seg_pos={0 0}, upload=133368]: Uploading piece...
Dec 10 23:43:43.451339742 (128h15m44.279048133s) [upload=133368]: Next returned nil segment
Dec 10 23:43:43.451352960 (128h15m44.279061351s) [upload=133368]: Waiting for error group managing segments...
Dec 10 23:43:43.455745583 (128h15m44.283453974s) [noise=true , address=45.91.116.13:4179 , piece_id=LELMW36GJQYKM7HQJSF2ODQGLAT7R67AMLG4MQPVUGOFSH6QUAGQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.456321180 (128h15m44.284029573s) [noise=true , address=45.91.116.10:4092 , piece_id=O2HDXSWQVNKGHZ7W5AWVPCGG3TJ4S4V7RLKT5XYGVHUBLYJHIZQQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.456419392 (128h15m44.284127782s) [noise=true , address=45.91.116.11:4133 , piece_id=I7J5DKPUGNGKQLDM74CYKNZXQUM7M3QQ4VCWMKPAFHPI6M756SNQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.457972823 (128h15m44.285681214s) [noise=true , address=145.79.227.45:4250 , piece_id=LMHSUWSONIYZ7JPVBNGHPQCJDIBZMMUZ6YNWQZHBOGU5QVHZ2Q6A, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.459053018 (128h15m44.286761412s) [noise=true , address=145.79.227.52:4466 , piece_id=AHS6KJVURYU5REAHEPTYQB4AKHJVDQUGLEFMLJV4M6RXLR46PMZQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.470388241 (128h15m44.298096635s) [noise=true , address=145.79.227.39:4047 , piece_id=6IPUBEPRB62GKJ3PZIFYPWC54TZVNPTZFYORNFGZWUIXPYC4EZQQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.471445505 (128h15m44.299153896s) [noise=true , address=145.79.227.2:4137 , piece_id=MD6LEPAJPQW5G4TWRHRKVPPZPCETLRPREVLLB5AYGU5PZANUOGDQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.475289360 (128h15m44.302997742s) [noise=true , address=45.91.116.9:4270 , piece_id=BTWWCBF64DXUORXC24CD2KG4VAMREXHZ4RSGCXAROLWGFO5VBQ6A, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.479327480 (128h15m44.307035871s) [noise=true , address=145.79.227.12:4798 , piece_id=4KRVETRCIKRCWA6CADS3YW5XFZM25IWYEOGCGXT563LAEJZWE7LQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:43.485225312 (128h15m44.312933689s) [noise=true , address=145.79.227.34:4937 , piece_id=3PX4TT5HLZW77T7SRYWWQGD24DCPPLLS5QRGG6DNSD7AYVSXDQLA, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:43:44.454989189 (128h15m45.282697592s) [noise=true , address=145.79.227.30:4794 , piece_id=7OLDNEYASADLBCX6EOIWLDZ24X366XGSQKVEUNX2TNZCH7KZDWBA, seg_pos={0 0}, upload=133368]: Done uploading piece. err: <nil>
Dec 10 23:44:43.285172169 (128h16m44.112880563s) [noise=true , address=77.90.31.10:4534 , piece_id=33CMFWHSKYSJUD4BX3CAT3FCSDD6YLGEALYJ3RFYLRDBE5X2SPRA, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:1266qFn3AGEr59ZmjnK4qg5q1s2iWD4UMTCR7tqsWP4rWfC8xWr, address:77.90.31.10:4534): protocol: write tcp 159.223.143.39:52682->77.90.31.10:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.285430757 (128h16m44.11313916s) [noise=true , address=77.90.31.19:4860 , piece_id=Q2U64XB7AK4IT35AR6OPOYS6LGYGUUGALYRO522URUUZRC6IGLUQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, address:77.90.31.19:4860): protocol: write tcp 159.223.143.39:38320->77.90.31.19:4860: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.285510735 (128h16m44.113219145s) [noise=true , address=77.90.31.16:4759 , piece_id=ID3XX6VXIRB5GAQHSQ6KEG6KYLPLJGSQULKUUUAGM6WBCNP7MVHA, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: write tcp 159.223.143.39:52328->77.90.31.16:4759: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.285712802 (128h16m44.113421193s) [noise=true , address=77.90.31.21:4910 , piece_id=5LDSTTLUR7V7WABWWMS6OD6ZWHRCVAN3W4BOOFTAZAABVN5V573Q, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:12onFkiP1wSRcNuzyqegL2EjfttSrQ5AMJwfp9PBn5urHQb3C2H, address:77.90.31.21:4910): protocol: write tcp 159.223.143.39:58196->77.90.31.21:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.285776948 (128h16m44.113485331s) [noise=true , address=77.90.31.14:4682 , piece_id=IMMATXQ23SDDXWSLXM73IIV7EAV4ZYIS23FLBYXQRDCNDTUCILQQ, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:1DMWgJrNMj9h9TDyiLiqFpfaa62DA6GQ8b8prUXoVCY63ERF4d, address:77.90.31.14:4682): protocol: write tcp 159.223.143.39:50148->77.90.31.14:4682: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.286026333 (128h16m44.113734726s) [noise=true , address=77.90.31.13:4653 , piece_id=AZSQXGFVP3WDU5URMNZLVAR3WIZWTDIKBNBYGD7PIYZP72CJK22A, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: write tcp 159.223.143.39:58294->77.90.31.13:4653: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.286104018 (128h16m44.113812403s) [noise=true , address=77.90.31.15:4712 , piece_id=TCWKRXUJJUQYGXVQDET4WUU5IMPCZ74A43KCJ2T3A5K6H57IQ4AA, seg_pos={0 0}, upload=133368]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: write tcp 159.223.143.39:58390->77.90.31.15:4712: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:43.286137038 (128h16m44.113845422s) [seg_pos={0 0}, upload=133368]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Dec 10 23:44:43.313980053 (128h16m44.141688444s) [upload=133368]: Done uploading segments...
Dec 10 23:44:43.314031298 (128h16m44.141739693s) [upload=133368]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 1m0.032035522s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................................XXX.X.......................] duration:
408.941493ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:44:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 408.971427ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................XXX.........................] duration:
544.537758ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:44:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 544.567019ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
17 out of 60 is failed (threshold=3) [.......................................XX..XXXXXX.XXXXXX.XXX] duration:
1m0.023134622s trace:
https://jaeger.dev.storj.io/trace/505bfcdf517cd080
FAIL (2025-12-10 23:44:12 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
Dec 10 23:43:12.491452203 (128h15m13.319160594s) [upload=133361]: Starting upload
Dec 10 23:43:12.491496750 (128h15m13.31920515s) [upload=133361]: Done starting upload
Dec 10 23:43:12.491601846 (128h15m13.319310223s) [upload=133361]: Uploading segments...
Dec 10 23:43:12.491793982 (128h15m13.319502359s) [upload=133361]: Got next segment. Inline: false
Dec 10 23:43:12.491844882 (128h15m13.319553274s) [upload=133361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:43:12.491853840 (128h15m13.319562237s) [upload=133361]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:43:12.518248663 (128h15m13.345957057s) [seg_pos={0 0}, upload=133361]: Begin upload segment...
Dec 10 23:43:12.518298266 (128h15m13.346006651s) [seg_pos={0 0}, upload=133361]: Done begin upload segment.
Dec 10 23:43:12.518326407 (128h15m13.346034805s) [noise=true , address=145.79.227.44:4199 , piece_id=BEX6FOQP5YR7OAA7VHZ52AIZ5BUE5Q5UCQS7L5SNNJSTRJQGRNNA, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.518349317 (128h15m13.346057713s) [noise=true , address=45.91.116.5:4328 , piece_id=CC3PXUKNFS2DLEI7WFBTYV7A7SI2ZWEBIZDCQUIHPPCWKY4GBR4A, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.519232536 (128h15m13.346940928s) [noise=true , address=145.79.227.47:4269 , piece_id=AU7CJ77NGK24OA5GE532PZ32VB6XYDQFJBJORIB7F7IQXCZPDACQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.519592583 (128h15m13.347300977s) [noise=true , address=45.91.116.10:4095 , piece_id=OOYOBTKMYOFTL3XCKV2ATP6Q4GD6LOR44DDSH6I2EAFF22T2N4QA, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.520072403 (128h15m13.347780869s) [noise=true , address=77.90.31.17:4790 , piece_id=MQ4KIFPMTPRM5OIF5NOXO53WPIT45ZB23A3G4OIGIJF2OIW64OGA, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.520633138 (128h15m13.348341535s) [noise=true , address=145.79.227.25:4625 , piece_id=FT3CNHPXJQW5JFNMRCA3OOYRGSTCW4SXBAU3V2WYDRFK6RKNFAAA, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.520797188 (128h15m13.348505584s) [noise=true , address=45.91.116.8:4071 , piece_id=NJ55MP2J5QLBRCJCBKIC4OFCHVS5YEU2O6CXRCYCS22VIJMO3DPQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.521317679 (128h15m13.349026073s) [noise=true , address=77.90.31.21:4906 , piece_id=SMQF7D6S26KT3LEHEP55CMGBRM6DYHJA4QF72A6W2VCYH3YAH7CQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.521560702 (128h15m13.349269081s) [noise=true , address=77.90.31.18:4815 , piece_id=A2DLPWRZNTTTLA3PYZB66FK6KQPOI3NNSQB4RM3Y74BLW4RL6ZXA, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.522849449 (128h15m13.350557846s) [noise=true , address=77.90.31.10:4541 , piece_id=NFLV3FQ3G23QHL3ZFCIVS4B5JF545OR7Q6YTMCRDRS6N475US22Q, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.523062541 (128h15m13.350770938s) [noise=true , address=77.90.31.14:4677 , piece_id=DRT2YDOS7PNYANUHLNNYMHTJPWS4W4NZSUULSSDFUGVCKD73K7DQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.523784348 (128h15m13.351492749s) [noise=true , address=45.91.116.7:4022 , piece_id=BFZPGTQPNYNQ4A7DXPUTZRE3GJ7JUWURXJQ5RNB7AAA2NT7X3ZPQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.523864988 (128h15m13.351573379s) [noise=true , address=145.79.227.11:4741 , piece_id=OWMQE5Y3POIYUJMRN6E7PMIOFOQDXK45H2RPH55GKAM3S54UMUWQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.524677520 (128h15m13.352385914s) [noise=true , address=77.90.31.19:4845 , piece_id=NB5V2KBL5SZHXILHFNAKUNTWDIR3ODL2MY7T42TVBT7RATFX2M3A, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.524982132 (128h15m13.352690526s) [noise=true , address=77.90.31.15:4709 , piece_id=B362WDHJ744OLGYVDDUXJ4XHJA5D76C77GRGVGY335UMNH3T6CEQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.525314609 (128h15m13.353023005s) [noise=true , address=145.79.227.33:4898 , piece_id=AAJU63EE37ZGN3BS2NNKIDHYSUSOVSGWJ6Y53ZWPTRTZGT673VEA, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.525786851 (128h15m13.353495248s) [noise=true , address=77.90.31.12:4606 , piece_id=KRCRYGCVTMT7ZK5SSZMEMOZM4XCCDGUMVPE4IKEMBRPILOZK35FQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.526597288 (128h15m13.354305693s) [noise=true , address=145.79.227.20:4459 , piece_id=A4ESGMD5ZRIS3I3H3FIXV7IQQUJIK6LGWLYI2EOWBYICP53VE2LQ, seg_pos={0 0}, upload=133361]: Uploading piece...
Dec 10 23:43:12.591717094 (128h15m13.419425474s) [upload=133361]: Next returned nil segment
Dec 10 23:43:12.591734216 (128h15m13.419442618s) [upload=133361]: Waiting for error group managing segments...
Dec 10 23:43:12.597075873 (128h15m13.424784266s) [noise=true , address=45.91.116.8:4071 , piece_id=NJ55MP2J5QLBRCJCBKIC4OFCHVS5YEU2O6CXRCYCS22VIJMO3DPQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.598167406 (128h15m13.4258758s) [noise=true , address=145.79.227.11:4741 , piece_id=OWMQE5Y3POIYUJMRN6E7PMIOFOQDXK45H2RPH55GKAM3S54UMUWQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.601416774 (128h15m13.429125166s) [noise=true , address=145.79.227.20:4459 , piece_id=A4ESGMD5ZRIS3I3H3FIXV7IQQUJIK6LGWLYI2EOWBYICP53VE2LQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.603045670 (128h15m13.430754062s) [noise=true , address=145.79.227.33:4898 , piece_id=AAJU63EE37ZGN3BS2NNKIDHYSUSOVSGWJ6Y53ZWPTRTZGT673VEA, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.610152546 (128h15m13.437860939s) [noise=true , address=145.79.227.25:4625 , piece_id=FT3CNHPXJQW5JFNMRCA3OOYRGSTCW4SXBAU3V2WYDRFK6RKNFAAA, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.610822743 (128h15m13.438531134s) [noise=true , address=45.91.116.10:4095 , piece_id=OOYOBTKMYOFTL3XCKV2ATP6Q4GD6LOR44DDSH6I2EAFF22T2N4QA, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.615142609 (128h15m13.442851002s) [noise=true , address=45.91.116.7:4022 , piece_id=BFZPGTQPNYNQ4A7DXPUTZRE3GJ7JUWURXJQ5RNB7AAA2NT7X3ZPQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:12.618314708 (128h15m13.446023099s) [noise=true , address=45.91.116.5:4328 , piece_id=CC3PXUKNFS2DLEI7WFBTYV7A7SI2ZWEBIZDCQUIHPPCWKY4GBR4A, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:13.538008870 (128h15m14.365717261s) [noise=true , address=145.79.227.44:4199 , piece_id=BEX6FOQP5YR7OAA7VHZ52AIZ5BUE5Q5UCQS7L5SNNJSTRJQGRNNA, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:43:14.170132525 (128h15m14.997840916s) [noise=true , address=145.79.227.47:4269 , piece_id=AU7CJ77NGK24OA5GE532PZ32VB6XYDQFJBJORIB7F7IQXCZPDACQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: <nil>
Dec 10 23:44:12.457488815 (128h16m13.285197212s) [noise=true , address=77.90.31.15:4709 , piece_id=B362WDHJ744OLGYVDDUXJ4XHJA5D76C77GRGVGY335UMNH3T6CEQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah, address:77.90.31.15:4709): protocol: rpc: dial tcp 77.90.31.15:4709: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.457689902 (128h16m13.285398303s) [noise=true , address=77.90.31.14:4677 , piece_id=DRT2YDOS7PNYANUHLNNYMHTJPWS4W4NZSUULSSDFUGVCKD73K7DQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, address:77.90.31.14:4677): protocol: rpc: dial tcp 77.90.31.14:4677: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.457714629 (128h16m13.285423031s) [noise=true , address=77.90.31.18:4815 , piece_id=A2DLPWRZNTTTLA3PYZB66FK6KQPOI3NNSQB4RM3Y74BLW4RL6ZXA, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw, address:77.90.31.18:4815): protocol: rpc: dial tcp 77.90.31.18:4815: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.458067290 (128h16m13.285775691s) [noise=true , address=77.90.31.17:4790 , piece_id=MQ4KIFPMTPRM5OIF5NOXO53WPIT45ZB23A3G4OIGIJF2OIW64OGA, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.458363134 (128h16m13.286071531s) [noise=true , address=77.90.31.12:4606 , piece_id=KRCRYGCVTMT7ZK5SSZMEMOZM4XCCDGUMVPE4IKEMBRPILOZK35FQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:1pLKmTja7ttT6vqjq8vHMYR9RLfnpqxY9xm9GDBG8ez9aHujs, address:77.90.31.12:4606): protocol: rpc: dial tcp 77.90.31.12:4606: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.458471227 (128h16m13.286179629s) [noise=true , address=77.90.31.10:4541 , piece_id=NFLV3FQ3G23QHL3ZFCIVS4B5JF545OR7Q6YTMCRDRS6N475US22Q, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: rpc: dial tcp 77.90.31.10:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.458552535 (128h16m13.286260932s) [noise=true , address=77.90.31.19:4845 , piece_id=NB5V2KBL5SZHXILHFNAKUNTWDIR3ODL2MY7T42TVBT7RATFX2M3A, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF, address:77.90.31.19:4845): protocol: rpc: dial tcp 77.90.31.19:4845: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.458728380 (128h16m13.286436773s) [noise=true , address=77.90.31.21:4906 , piece_id=SMQF7D6S26KT3LEHEP55CMGBRM6DYHJA4QF72A6W2VCYH3YAH7CQ, seg_pos={0 0}, upload=133361]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:44:12.458756983 (128h16m13.286465368s) [seg_pos={0 0}, upload=133361]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Dec 10 23:44:12.478726475 (128h16m13.306434867s) [upload=133361]: Done uploading segments...
Dec 10 23:44:12.478785313 (128h16m13.306493714s) [upload=133361]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.023148691s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................X....X...X........X...X......] duration:
471.965202ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:44:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.990687ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................XXX........................] duration:
501.161149ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:43:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 501.192545ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................................XXX.X......................] duration:
404.280806ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:43:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.329578ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
16 out of 60 is failed (threshold=3) [........................................XX..XXXXXX.XXXXXX.XX] duration:
1m0.022690613s trace:
https://jaeger.dev.storj.io/trace/4cf850ee8c6f43ec
FAIL (2025-12-10 23:43:12 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
Dec 10 23:42:12.420950691 (128h14m13.248659129s) [upload=133346]: Starting upload
Dec 10 23:42:12.421051698 (128h14m13.248760091s) [upload=133346]: Done starting upload
Dec 10 23:42:12.421146798 (128h14m13.248855175s) [upload=133346]: Uploading segments...
Dec 10 23:42:12.421442132 (128h14m13.249150535s) [upload=133346]: Got next segment. Inline: false
Dec 10 23:42:12.421490539 (128h14m13.249198937s) [upload=133346]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:42:12.421496398 (128h14m13.249204799s) [upload=133346]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:42:12.450424472 (128h14m13.278132866s) [seg_pos={0 0}, upload=133346]: Begin upload segment...
Dec 10 23:42:12.450491938 (128h14m13.278200333s) [seg_pos={0 0}, upload=133346]: Done begin upload segment.
Dec 10 23:42:12.450523837 (128h14m13.278232231s) [noise=true , address=77.90.31.16:4754 , piece_id=HGAKAZWPTW2DKGLI6T3OKH2TIUDI2RV5BESWDQNCOVQGKPFMH7LQ, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.451182555 (128h14m13.278890964s) [noise=true , address=145.79.227.25:4630 , piece_id=E3WCEPJPYMGKTMJC6GBFPDJDWKWMANRFRJDVOVUZH3YOTO3LK56A, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.451983253 (128h14m13.279691651s) [noise=true , address=145.79.227.28:4745 , piece_id=EKULWXDLAYWKZAPS4VEFGN2S3XGDZSLEAGLX6BUPZ2OK2JM5MJ6A, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.452527624 (128h14m13.280236025s) [noise=true , address=145.79.227.48:4308 , piece_id=7KTWLFOZOZ2SRRNZ2YY4TH6TGA4XDDINYOGABJJYJVBZIQF22LEA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.453080194 (128h14m13.280788595s) [noise=true , address=145.79.227.33:4883 , piece_id=DTWORITU7XVYH7MX35MVIS6GL42KELGNZY5FE25LBIAFORFJSRVQ, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.453344597 (128h14m13.281052998s) [noise=true , address=45.91.116.5:4314 , piece_id=EVM2JBWZJWAVNZKKVDJHDTO7GCPIXYHSFAZRPWNZPY4USMCF4MXQ, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.453985831 (128h14m13.281694233s) [noise=true , address=77.90.31.13:4654 , piece_id=V72SGUBSNH7MAI7EL66G66WAGFRETWQDWOE7VNKEI6IJMU3BG3NA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.454294210 (128h14m13.282002591s) [noise=true , address=77.90.31.17:4789 , piece_id=2UYSOZG36IM2V2J3FUR6FO2WBHKP7OCH52L7CPTYACIVYB5BLHTA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.454827314 (128h14m13.282535715s) [noise=true , address=77.90.31.21:4886 , piece_id=2OWLBAJELI6STW7AQ7HZQ5NRYGYZ22ORI65PLD6X6OKTNUWXB7YA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.455221793 (128h14m13.282930195s) [noise=true , address=45.91.116.7:4040 , piece_id=GBWFMW4TLFQ3CRCQWQU36LJOM3FW4INTNXYEIVSKROG57VU5CTDA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.455863478 (128h14m13.283571875s) [noise=true , address=77.90.31.21:4893 , piece_id=RPTMWCSPMTYJBODP4UOB5X2OINLFA224LJGZ6IJA6AZUSWKSWADA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.455890613 (128h14m13.283599018s) [noise=true , address=145.79.227.36:4528 , piece_id=I4C3ND5W6DLJLQ2VEKZTQPXR73C7T5SDOIL6LT7IWN2EUBOZYU4A, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.456429492 (128h14m13.284137895s) [noise=true , address=145.79.227.30:4779 , piece_id=AVU7UTI5WI6ZYPP4BT2IDAAYRIHQCRR6NDAHSHZLAZCCAZ4RBH5Q, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.456472911 (128h14m13.284181312s) [noise=true , address=77.90.31.12:4594 , piece_id=JY5VF2RYJ37X7C2NBJYB6LUXUNHZ7XT4MXCLIMAYVTGJQOP5S2AQ, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.456944935 (128h14m13.284653337s) [noise=true , address=145.79.227.45:4245 , piece_id=JIPLBN3EQG56UGNS6ZDCU3JRECJUGKEW45XCNENDDV2WWE74FUTA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.457090897 (128h14m13.284799298s) [noise=true , address=77.90.31.18:4810 , piece_id=NDVVDLV4CQRE3Q4J3PNWXHPEKSMRQVQ53JTODFHA6PQAH5JPFGSA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.459520823 (128h14m13.287229207s) [noise=true , address=77.90.31.10:4552 , piece_id=7MAA2LQQDGONJFH2XGSMLB73AWUNADQXL3IGB5I37JALXKTBNWAA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.460726206 (128h14m13.288434583s) [noise=true , address=77.90.31.11:4568 , piece_id=K27WICLMDQJYG24FLCAHI2VMFZL57UMU6F2QW7U2JGNGE2267WSA, seg_pos={0 0}, upload=133346]: Uploading piece...
Dec 10 23:42:12.522282125 (128h14m13.349990516s) [upload=133346]: Next returned nil segment
Dec 10 23:42:12.522293974 (128h14m13.350002366s) [upload=133346]: Waiting for error group managing segments...
Dec 10 23:42:12.528775151 (128h14m13.356483544s) [noise=true , address=145.79.227.48:4308 , piece_id=7KTWLFOZOZ2SRRNZ2YY4TH6TGA4XDDINYOGABJJYJVBZIQF22LEA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.528920235 (128h14m13.356628631s) [noise=true , address=145.79.227.25:4630 , piece_id=E3WCEPJPYMGKTMJC6GBFPDJDWKWMANRFRJDVOVUZH3YOTO3LK56A, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.532338928 (128h14m13.36004732s) [noise=true , address=145.79.227.33:4883 , piece_id=DTWORITU7XVYH7MX35MVIS6GL42KELGNZY5FE25LBIAFORFJSRVQ, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.532651469 (128h14m13.36035986s) [noise=true , address=145.79.227.36:4528 , piece_id=I4C3ND5W6DLJLQ2VEKZTQPXR73C7T5SDOIL6LT7IWN2EUBOZYU4A, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.537612669 (128h14m13.36532106s) [noise=true , address=145.79.227.45:4245 , piece_id=JIPLBN3EQG56UGNS6ZDCU3JRECJUGKEW45XCNENDDV2WWE74FUTA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.538783395 (128h14m13.36649179s) [noise=true , address=45.91.116.5:4314 , piece_id=EVM2JBWZJWAVNZKKVDJHDTO7GCPIXYHSFAZRPWNZPY4USMCF4MXQ, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.539394830 (128h14m13.367103221s) [noise=true , address=45.91.116.7:4040 , piece_id=GBWFMW4TLFQ3CRCQWQU36LJOM3FW4INTNXYEIVSKROG57VU5CTDA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:12.547468927 (128h14m13.375177326s) [noise=true , address=145.79.227.28:4745 , piece_id=EKULWXDLAYWKZAPS4VEFGN2S3XGDZSLEAGLX6BUPZ2OK2JM5MJ6A, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:42:13.475194746 (128h14m14.302903145s) [noise=true , address=145.79.227.30:4779 , piece_id=AVU7UTI5WI6ZYPP4BT2IDAAYRIHQCRR6NDAHSHZLAZCCAZ4RBH5Q, seg_pos={0 0}, upload=133346]: Done uploading piece. err: <nil>
Dec 10 23:43:12.379195195 (128h15m13.206903591s) [noise=true , address=77.90.31.16:4754 , piece_id=HGAKAZWPTW2DKGLI6T3OKH2TIUDI2RV5BESWDQNCOVQGKPFMH7LQ, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:12KKCnEfLWFbLY7ViZ8uzgKfNDeVby4XaoRiBKiQCH39mq82RJ8, address:77.90.31.16:4754): protocol: rpc: dial tcp 77.90.31.16:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.379303010 (128h15m13.20701141s) [noise=true , address=77.90.31.13:4654 , piece_id=V72SGUBSNH7MAI7EL66G66WAGFRETWQDWOE7VNKEI6IJMU3BG3NA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: rpc: dial tcp 77.90.31.13:4654: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.379469956 (128h15m13.207178351s) [noise=true , address=77.90.31.11:4568 , piece_id=K27WICLMDQJYG24FLCAHI2VMFZL57UMU6F2QW7U2JGNGE2267WSA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:1H1L8DTKD7W7Xx1PVT7KzB9GkGBadnTEj9BHtbSgcW1S7Jj8Tw, address:77.90.31.11:4568): protocol: rpc: dial tcp 77.90.31.11:4568: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.379603917 (128h15m13.207312318s) [noise=true , address=77.90.31.21:4886 , piece_id=2OWLBAJELI6STW7AQ7HZQ5NRYGYZ22ORI65PLD6X6OKTNUWXB7YA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:12VMSq3AX3PYjchs8n7yDWVWdKVRUVoe3PiyDitkX1EwSUwsTeM, address:77.90.31.21:4886): protocol: rpc: dial tcp 77.90.31.21:4886: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.379692561 (128h15m13.207400955s) [noise=true , address=77.90.31.17:4789 , piece_id=2UYSOZG36IM2V2J3FUR6FO2WBHKP7OCH52L7CPTYACIVYB5BLHTA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: rpc: dial tcp 77.90.31.17:4789: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.379862818 (128h15m13.207571218s) [noise=true , address=77.90.31.12:4594 , piece_id=JY5VF2RYJ37X7C2NBJYB6LUXUNHZ7XT4MXCLIMAYVTGJQOP5S2AQ, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: rpc: dial tcp 77.90.31.12:4594: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.379952666 (128h15m13.207661067s) [noise=true , address=77.90.31.21:4893 , piece_id=RPTMWCSPMTYJBODP4UOB5X2OINLFA224LJGZ6IJA6AZUSWKSWADA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC, address:77.90.31.21:4893): protocol: rpc: dial tcp 77.90.31.21:4893: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.380211627 (128h15m13.207920021s) [noise=true , address=77.90.31.10:4552 , piece_id=7MAA2LQQDGONJFH2XGSMLB73AWUNADQXL3IGB5I37JALXKTBNWAA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:1sZq8qBrujZ1JMFYMhfCLsjnm7V3FE2KVxPuqFeWC2VMW1yzfp, address:77.90.31.10:4552): protocol: rpc: dial tcp 77.90.31.10:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.380359528 (128h15m13.208067928s) [noise=true , address=77.90.31.18:4810 , piece_id=NDVVDLV4CQRE3Q4J3PNWXHPEKSMRQVQ53JTODFHA6PQAH5JPFGSA, seg_pos={0 0}, upload=133346]: Done uploading piece. err: ecclient: upload failed (node:12mfG3ZeKyFWEPsTgSoR4wqiUYY7YobwmMtsg1GsCorquCZLX5F, address:77.90.31.18:4810): protocol: write tcp 159.223.143.39:37658->77.90.31.18:4810: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:43:12.380401091 (128h15m13.208109487s) [seg_pos={0 0}, upload=133346]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Dec 10 23:43:12.400385552 (128h15m13.228093943s) [upload=133346]: Done uploading segments...
Dec 10 23:43:12.400420056 (128h15m13.228128451s) [upload=133346]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.022714934s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................X....X...X........X...X.....] duration:
460.801658ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:43:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.83422ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration:
529.652269ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:42:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 529.67722ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................................XXX.X.....................] duration:
427.222601ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:42:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 427.51102ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [........................................XXXXXXX.XXXXXXXXXXX.] duration:
1.536908102s trace:
https://jaeger.dev.storj.io/trace/681e78212698f7a
FAIL (2025-12-10 23:42:32 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
false
Dec 10 23:41:31.031122655 (128h13m31.858831052s) [upload=133334]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:41:31.031131852 (128h13m31.858840247s) [upload=133334]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:41:31.059564350 (128h13m31.887272761s) [seg_pos={0 0}, upload=133334]: Begin upload segment...
Dec 10 23:41:31.059650313 (128h13m31.887358705s) [seg_pos={0 0}, upload=133334]: Done begin upload segment.
Dec 10 23:41:31.059705751 (128h13m31.88741415s) [noise=true , address=77.90.31.19:4868 , piece_id=5EFXZ4BB7E37D7WYSQ7HLW25FCIS66UID35K5PJJ6OTFLEJFZTYQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.066172597 (128h13m31.893880994s) [noise=true , address=77.90.31.11:4584 , piece_id=ACZDACJI3ZPCMHN43YPO6PWNENCU3HGLXYBMS5FJKAUOV5FDONMQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.067394398 (128h13m31.89510279s) [noise=true , address=77.90.31.16:4739 , piece_id=RFRW65EWAOMO7DR4AD5KJBZD75F67LLO66ZLVVLOSAH6RFSLDOFQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.069713583 (128h13m31.897421982s) [noise=true , address=45.91.116.13:4165 , piece_id=7SHLFAJJ2VQE6LW5YGHZQP47EG3PWZKMGQ6DTQWSI5DHCUFUAIYQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.070121691 (128h13m31.897830085s) [noise=true , address=77.90.31.12:4600 , piece_id=LHB7J2GC5XA7UQDOPYBOLPKFYHBTLSKQUYKTP3GTCKPXRVZV4SIQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.072403874 (128h13m31.900112271s) [noise=true , address=77.90.31.14:4676 , piece_id=RNFDTZNBW6TMO2OBMRZGBVB3OBVFICS6DTO5QYJGCF7ZB6JDY7FQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.073170167 (128h13m31.900878564s) [noise=true , address=145.79.227.36:4525 , piece_id=XQH46K4ULVTLJ4YT25GY6PE7X7GAEQ64YJX7KUITXOZIWUBD7AMQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.074357541 (128h13m31.902065935s) [noise=true , address=45.91.116.11:4123 , piece_id=2UJMI7SYKE4Y42JJ5D7V5QTZZM2X56ZH5TZP7GXBCG4P7K3JT4LQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.075753533 (128h13m31.90346191s) [noise=true , address=5.231.73.4:4018 , piece_id=YXKUKVGKD26PWDEAD3SQVSKW5EOWPPJUBPZHHPGPEICCHNTX3D5Q, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.076979886 (128h13m31.904688283s) [noise=true , address=145.79.227.25:4626 , piece_id=2RRCKQQGASHVC7222AO7AOSRUTYMYY4YYIMPSF3QW4NCRZGJD2BA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.078305282 (128h13m31.906013685s) [noise=true , address=77.90.31.17:4787 , piece_id=HHELYHXTJX46KOKOE46E3AI6S5UFJEVZYYIUTX725BNOFR4HUSJQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.079607670 (128h13m31.90731605s) [noise=true , address=45.91.116.14:4206 , piece_id=Z2KSKNYVQWARKPJ5BJZ42PMYECCIQPIKHDI35656FVOJZ7TYMJ7Q, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.079710829 (128h13m31.907419211s) [noise=true , address=145.79.227.42:4155 , piece_id=J6M7BQMRKYR2WGUV5TOOYPT63C2OANZKH37TDPQ5OCQ7DQ6EIKBA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.081137295 (128h13m31.908845692s) [noise=true , address=145.79.227.22:4542 , piece_id=EPO4XT3ETOPJI26IGIVVXWECSWUJZ6CHSTFVK6REKTESIHCCBS2A, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.081159774 (128h13m31.908868169s) [noise=true , address=45.91.116.5:4325 , piece_id=O72XOZJPMF2LRZHAAIACMFTNN26RJHEXVKZ6ZBPVEZ7H5JTB4BQA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.081218376 (128h13m31.90892677s) [noise=true , address=45.91.116.6:4984 , piece_id=RAWXFSM6OE6CJGKNQ3KLIJXR7KAM76O7WJ5SEYCQGKLVIYQNB7HQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.082499978 (128h13m31.910208431s) [noise=true , address=45.91.116.7:4020 , piece_id=Y3PRSEWTHBIOZ4NFMG7TAWEYZCWIDPYIQAMVYSLE37O5MVTKTZMQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.083781729 (128h13m31.911490127s) [noise=true , address=145.79.227.31:4828 , piece_id=EYOT7CT765EAFW2BREM2TRVPP6K4J5OVE4RU3S2DS7BKSZ7EHVNA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.131993193 (128h13m31.959701585s) [upload=133334]: Next returned nil segment
Dec 10 23:41:31.132006506 (128h13m31.959714888s) [upload=133334]: Waiting for error group managing segments...
Dec 10 23:41:31.135430305 (128h13m31.963138686s) [noise=true , address=45.91.116.13:4165 , piece_id=7SHLFAJJ2VQE6LW5YGHZQP47EG3PWZKMGQ6DTQWSI5DHCUFUAIYQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.135510169 (128h13m31.963218561s) [noise=true , address=45.91.116.7:4020 , piece_id=Y3PRSEWTHBIOZ4NFMG7TAWEYZCWIDPYIQAMVYSLE37O5MVTKTZMQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.136217563 (128h13m31.963925954s) [noise=true , address=45.91.116.6:4984 , piece_id=RAWXFSM6OE6CJGKNQ3KLIJXR7KAM76O7WJ5SEYCQGKLVIYQNB7HQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.149455289 (128h13m31.977163682s) [noise=true , address=45.91.116.11:4123 , piece_id=2UJMI7SYKE4Y42JJ5D7V5QTZZM2X56ZH5TZP7GXBCG4P7K3JT4LQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.150598913 (128h13m31.978307306s) [noise=true , address=45.91.116.14:4206 , piece_id=Z2KSKNYVQWARKPJ5BJZ42PMYECCIQPIKHDI35656FVOJZ7TYMJ7Q, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.150850187 (128h13m31.978558578s) [noise=true , address=45.91.116.5:4325 , piece_id=O72XOZJPMF2LRZHAAIACMFTNN26RJHEXVKZ6ZBPVEZ7H5JTB4BQA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.155768838 (128h13m31.983477229s) [noise=true , address=145.79.227.36:4525 , piece_id=XQH46K4ULVTLJ4YT25GY6PE7X7GAEQ64YJX7KUITXOZIWUBD7AMQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.156382975 (128h13m31.984091371s) [noise=true , address=145.79.227.22:4542 , piece_id=EPO4XT3ETOPJI26IGIVVXWECSWUJZ6CHSTFVK6REKTESIHCCBS2A, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.158609296 (128h13m31.986317696s) [noise=true , address=145.79.227.42:4155 , piece_id=J6M7BQMRKYR2WGUV5TOOYPT63C2OANZKH37TDPQ5OCQ7DQ6EIKBA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.163914850 (128h13m31.99162323s) [noise=true , address=145.79.227.25:4626 , piece_id=2RRCKQQGASHVC7222AO7AOSRUTYMYY4YYIMPSF3QW4NCRZGJD2BA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.164483114 (128h13m31.992191514s) [noise=true , address=145.79.227.31:4828 , piece_id=EYOT7CT765EAFW2BREM2TRVPP6K4J5OVE4RU3S2DS7BKSZ7EHVNA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.252461347 (128h13m32.080169739s) [noise=true , address=5.231.73.4:4018 , piece_id=YXKUKVGKD26PWDEAD3SQVSKW5EOWPPJUBPZHHPGPEICCHNTX3D5Q, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:42:30.985348031 (128h14m31.813056423s) [noise=true , address=77.90.31.12:4600 , piece_id=LHB7J2GC5XA7UQDOPYBOLPKFYHBTLSKQUYKTP3GTCKPXRVZV4SIQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.985574256 (128h14m31.81328266s) [noise=true , address=77.90.31.17:4787 , piece_id=HHELYHXTJX46KOKOE46E3AI6S5UFJEVZYYIUTX725BNOFR4HUSJQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 159.223.143.39:56078->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.985636024 (128h14m31.813344403s) [noise=true , address=77.90.31.19:4868 , piece_id=5EFXZ4BB7E37D7WYSQ7HLW25FCIS66UID35K5PJJ6OTFLEJFZTYQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.985747339 (128h14m31.813455744s) [noise=true , address=77.90.31.14:4676 , piece_id=RNFDTZNBW6TMO2OBMRZGBVB3OBVFICS6DTO5QYJGCF7ZB6JDY7FQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:1tpRh8Xvyxe72LDgR9rkUFx9qV88axb9DUYDhRnrzST2cQuHqC, address:77.90.31.14:4676): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:42:30.985953823 (128h14m31.81366222s) [noise=true , address=77.90.31.11:4584 , piece_id=ACZDACJI3ZPCMHN43YPO6PWNENCU3HGLXYBMS5FJKAUOV5FDONMQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.986291293 (128h14m31.813999695s) [noise=true , address=77.90.31.16:4739 , piece_id=RFRW65EWAOMO7DR4AD5KJBZD75F67LLO66ZLVVLOSAH6RFSLDOFQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:42:30.986418000 (128h14m31.814126389s) [seg_pos={0 0}, upload=133334]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:42:31.007548197 (128h14m31.835256609s) [upload=133334]: Done uploading segments...
Dec 10 23:42:31.007584801 (128h14m31.835293255s) [upload=133334]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1.536926014s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [.........................................XXXXXXX.XXXXXXXXXXX] duration:
1m0.023692128s trace:
https://jaeger.dev.storj.io/trace/681e78212698f7a
FAIL (2025-12-10 23:42:31 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
false
Dec 10 23:41:31.031122655 (128h13m31.858831052s) [upload=133334]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:41:31.031131852 (128h13m31.858840247s) [upload=133334]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:41:31.059564350 (128h13m31.887272761s) [seg_pos={0 0}, upload=133334]: Begin upload segment...
Dec 10 23:41:31.059650313 (128h13m31.887358705s) [seg_pos={0 0}, upload=133334]: Done begin upload segment.
Dec 10 23:41:31.059705751 (128h13m31.88741415s) [noise=true , address=77.90.31.19:4868 , piece_id=5EFXZ4BB7E37D7WYSQ7HLW25FCIS66UID35K5PJJ6OTFLEJFZTYQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.066172597 (128h13m31.893880994s) [noise=true , address=77.90.31.11:4584 , piece_id=ACZDACJI3ZPCMHN43YPO6PWNENCU3HGLXYBMS5FJKAUOV5FDONMQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.067394398 (128h13m31.89510279s) [noise=true , address=77.90.31.16:4739 , piece_id=RFRW65EWAOMO7DR4AD5KJBZD75F67LLO66ZLVVLOSAH6RFSLDOFQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.069713583 (128h13m31.897421982s) [noise=true , address=45.91.116.13:4165 , piece_id=7SHLFAJJ2VQE6LW5YGHZQP47EG3PWZKMGQ6DTQWSI5DHCUFUAIYQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.070121691 (128h13m31.897830085s) [noise=true , address=77.90.31.12:4600 , piece_id=LHB7J2GC5XA7UQDOPYBOLPKFYHBTLSKQUYKTP3GTCKPXRVZV4SIQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.072403874 (128h13m31.900112271s) [noise=true , address=77.90.31.14:4676 , piece_id=RNFDTZNBW6TMO2OBMRZGBVB3OBVFICS6DTO5QYJGCF7ZB6JDY7FQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.073170167 (128h13m31.900878564s) [noise=true , address=145.79.227.36:4525 , piece_id=XQH46K4ULVTLJ4YT25GY6PE7X7GAEQ64YJX7KUITXOZIWUBD7AMQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.074357541 (128h13m31.902065935s) [noise=true , address=45.91.116.11:4123 , piece_id=2UJMI7SYKE4Y42JJ5D7V5QTZZM2X56ZH5TZP7GXBCG4P7K3JT4LQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.075753533 (128h13m31.90346191s) [noise=true , address=5.231.73.4:4018 , piece_id=YXKUKVGKD26PWDEAD3SQVSKW5EOWPPJUBPZHHPGPEICCHNTX3D5Q, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.076979886 (128h13m31.904688283s) [noise=true , address=145.79.227.25:4626 , piece_id=2RRCKQQGASHVC7222AO7AOSRUTYMYY4YYIMPSF3QW4NCRZGJD2BA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.078305282 (128h13m31.906013685s) [noise=true , address=77.90.31.17:4787 , piece_id=HHELYHXTJX46KOKOE46E3AI6S5UFJEVZYYIUTX725BNOFR4HUSJQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.079607670 (128h13m31.90731605s) [noise=true , address=45.91.116.14:4206 , piece_id=Z2KSKNYVQWARKPJ5BJZ42PMYECCIQPIKHDI35656FVOJZ7TYMJ7Q, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.079710829 (128h13m31.907419211s) [noise=true , address=145.79.227.42:4155 , piece_id=J6M7BQMRKYR2WGUV5TOOYPT63C2OANZKH37TDPQ5OCQ7DQ6EIKBA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.081137295 (128h13m31.908845692s) [noise=true , address=145.79.227.22:4542 , piece_id=EPO4XT3ETOPJI26IGIVVXWECSWUJZ6CHSTFVK6REKTESIHCCBS2A, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.081159774 (128h13m31.908868169s) [noise=true , address=45.91.116.5:4325 , piece_id=O72XOZJPMF2LRZHAAIACMFTNN26RJHEXVKZ6ZBPVEZ7H5JTB4BQA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.081218376 (128h13m31.90892677s) [noise=true , address=45.91.116.6:4984 , piece_id=RAWXFSM6OE6CJGKNQ3KLIJXR7KAM76O7WJ5SEYCQGKLVIYQNB7HQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.082499978 (128h13m31.910208431s) [noise=true , address=45.91.116.7:4020 , piece_id=Y3PRSEWTHBIOZ4NFMG7TAWEYZCWIDPYIQAMVYSLE37O5MVTKTZMQ, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.083781729 (128h13m31.911490127s) [noise=true , address=145.79.227.31:4828 , piece_id=EYOT7CT765EAFW2BREM2TRVPP6K4J5OVE4RU3S2DS7BKSZ7EHVNA, seg_pos={0 0}, upload=133334]: Uploading piece...
Dec 10 23:41:31.131993193 (128h13m31.959701585s) [upload=133334]: Next returned nil segment
Dec 10 23:41:31.132006506 (128h13m31.959714888s) [upload=133334]: Waiting for error group managing segments...
Dec 10 23:41:31.135430305 (128h13m31.963138686s) [noise=true , address=45.91.116.13:4165 , piece_id=7SHLFAJJ2VQE6LW5YGHZQP47EG3PWZKMGQ6DTQWSI5DHCUFUAIYQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.135510169 (128h13m31.963218561s) [noise=true , address=45.91.116.7:4020 , piece_id=Y3PRSEWTHBIOZ4NFMG7TAWEYZCWIDPYIQAMVYSLE37O5MVTKTZMQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.136217563 (128h13m31.963925954s) [noise=true , address=45.91.116.6:4984 , piece_id=RAWXFSM6OE6CJGKNQ3KLIJXR7KAM76O7WJ5SEYCQGKLVIYQNB7HQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.149455289 (128h13m31.977163682s) [noise=true , address=45.91.116.11:4123 , piece_id=2UJMI7SYKE4Y42JJ5D7V5QTZZM2X56ZH5TZP7GXBCG4P7K3JT4LQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.150598913 (128h13m31.978307306s) [noise=true , address=45.91.116.14:4206 , piece_id=Z2KSKNYVQWARKPJ5BJZ42PMYECCIQPIKHDI35656FVOJZ7TYMJ7Q, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.150850187 (128h13m31.978558578s) [noise=true , address=45.91.116.5:4325 , piece_id=O72XOZJPMF2LRZHAAIACMFTNN26RJHEXVKZ6ZBPVEZ7H5JTB4BQA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.155768838 (128h13m31.983477229s) [noise=true , address=145.79.227.36:4525 , piece_id=XQH46K4ULVTLJ4YT25GY6PE7X7GAEQ64YJX7KUITXOZIWUBD7AMQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.156382975 (128h13m31.984091371s) [noise=true , address=145.79.227.22:4542 , piece_id=EPO4XT3ETOPJI26IGIVVXWECSWUJZ6CHSTFVK6REKTESIHCCBS2A, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.158609296 (128h13m31.986317696s) [noise=true , address=145.79.227.42:4155 , piece_id=J6M7BQMRKYR2WGUV5TOOYPT63C2OANZKH37TDPQ5OCQ7DQ6EIKBA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.163914850 (128h13m31.99162323s) [noise=true , address=145.79.227.25:4626 , piece_id=2RRCKQQGASHVC7222AO7AOSRUTYMYY4YYIMPSF3QW4NCRZGJD2BA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.164483114 (128h13m31.992191514s) [noise=true , address=145.79.227.31:4828 , piece_id=EYOT7CT765EAFW2BREM2TRVPP6K4J5OVE4RU3S2DS7BKSZ7EHVNA, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:41:31.252461347 (128h13m32.080169739s) [noise=true , address=5.231.73.4:4018 , piece_id=YXKUKVGKD26PWDEAD3SQVSKW5EOWPPJUBPZHHPGPEICCHNTX3D5Q, seg_pos={0 0}, upload=133334]: Done uploading piece. err: <nil>
Dec 10 23:42:30.985348031 (128h14m31.813056423s) [noise=true , address=77.90.31.12:4600 , piece_id=LHB7J2GC5XA7UQDOPYBOLPKFYHBTLSKQUYKTP3GTCKPXRVZV4SIQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:12PKT2EcU6jAe6oUUdiTcwxGQsFa36uEmown77SemqQMM5Xw5nc, address:77.90.31.12:4600): protocol: rpc: dial tcp 77.90.31.12:4600: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.985574256 (128h14m31.81328266s) [noise=true , address=77.90.31.17:4787 , piece_id=HHELYHXTJX46KOKOE46E3AI6S5UFJEVZYYIUTX725BNOFR4HUSJQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 159.223.143.39:56078->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.985636024 (128h14m31.813344403s) [noise=true , address=77.90.31.19:4868 , piece_id=5EFXZ4BB7E37D7WYSQ7HLW25FCIS66UID35K5PJJ6OTFLEJFZTYQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: rpc: dial tcp 77.90.31.19:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.985747339 (128h14m31.813455744s) [noise=true , address=77.90.31.14:4676 , piece_id=RNFDTZNBW6TMO2OBMRZGBVB3OBVFICS6DTO5QYJGCF7ZB6JDY7FQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:1tpRh8Xvyxe72LDgR9rkUFx9qV88axb9DUYDhRnrzST2cQuHqC, address:77.90.31.14:4676): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:42:30.985953823 (128h14m31.81366222s) [noise=true , address=77.90.31.11:4584 , piece_id=ACZDACJI3ZPCMHN43YPO6PWNENCU3HGLXYBMS5FJKAUOV5FDONMQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:30.986291293 (128h14m31.813999695s) [noise=true , address=77.90.31.16:4739 , piece_id=RFRW65EWAOMO7DR4AD5KJBZD75F67LLO66ZLVVLOSAH6RFSLDOFQ, seg_pos={0 0}, upload=133334]: Done uploading piece. err: ecclient: upload failed (node:129B3N7yjJvwCzoVLTqWsPbcLQ5TQUyAzg1RCkbSAVKPWYTG8eb, address:77.90.31.16:4739): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:42:30.986418000 (128h14m31.814126389s) [seg_pos={0 0}, upload=133334]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Dec 10 23:42:31.007548197 (128h14m31.835256609s) [upload=133334]: Done uploading segments...
Dec 10 23:42:31.007584801 (128h14m31.835293255s) [upload=133334]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 1m0.023708856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................X....X...X........X...X....] duration:
430.535668ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:42:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 430.564017ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
15 out of 60 is failed (threshold=3) [.........................................XX..XXXXXX.XXXXXX.X] duration:
1m0.025860704s trace:
https://jaeger.dev.storj.io/trace/5115a82f9e00d49b
FAIL (2025-12-10 23:42:09 UTC)
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 9)
======LOGS======
Dec 10 23:41:09.410130540 (128h13m10.237838935s) [upload=133328]: Starting upload
Dec 10 23:41:09.410192655 (128h13m10.23790109s) [upload=133328]: Done starting upload
Dec 10 23:41:09.410316296 (128h13m10.238024673s) [upload=133328]: Uploading segments...
Dec 10 23:41:09.410607215 (128h13m10.238315594s) [upload=133328]: Got next segment. Inline: false
Dec 10 23:41:09.410675579 (128h13m10.238383983s) [upload=133328]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 10 23:41:09.410681826 (128h13m10.238390218s) [upload=133328]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 10 23:41:09.438101558 (128h13m10.265809936s) [seg_pos={0 0}, upload=133328]: Begin upload segment...
Dec 10 23:41:09.438150557 (128h13m10.265858934s) [seg_pos={0 0}, upload=133328]: Done begin upload segment.
Dec 10 23:41:09.438187800 (128h13m10.265896179s) [noise=true , address=64.49.13.169:4709 , piece_id=VWM5U5CPKOYY3UDHLVKHL6V3ZCD5GXALKVPRBO37LUUZILGOSXZA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.438207191 (128h13m10.265915592s) [noise=true , address=145.79.227.4:4193 , piece_id=LNKONJYHPKJ2J2EMEDVV5WOETCFWMKQOKS4GPWIEMW3EYRDHYVZQ, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.438738617 (128h13m10.266447009s) [noise=true , address=77.90.31.10:4530 , piece_id=WOWZQTYQRI3GH7JMGCJU43SNQYSKUJ5ZWLH3YUVDLM3C2OJMB7QA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.439068677 (128h13m10.266777071s) [noise=true , address=45.91.116.6:4984 , piece_id=PEPDXX3MKFARABFQKZ2AGWNA7LLX5W7DFEZQDA34B7FTC6TRSAEA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.439524157 (128h13m10.267232551s) [noise=true , address=5.231.73.5:4004 , piece_id=26RTHLRR5HPKIXCLIOKL4WXUK3J2A3EUE6A25FXCHOT3LPPTSSPA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.440057915 (128h13m10.267766308s) [noise=true , address=64.49.13.170:4725 , piece_id=AKA4YLYUG4NK2OC5PM3XWGMZI7VG34T6ZTAOELCYYEL6NQZYFXZA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.440664017 (128h13m10.268372411s) [noise=true , address=77.90.31.18:4830 , piece_id=7ETI4WFQJM2FIYS2H4BLHA7AYDP7HHY3SGVUBRJXLBBDMJUQOY7Q, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.441204978 (128h13m10.268913369s) [noise=true , address=77.90.31.14:4694 , piece_id=OJBGBS6OLLZN5WTQR2WFLKYPKCA2J66NAHNGXFOTWRI5NLXAQRWA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.441271521 (128h13m10.268979917s) [noise=true , address=77.90.31.21:4896 , piece_id=6QAN2QZQ2SZU4FUIZHBRCCVP4EUTKMK4GDILUJVHZCPRDKCBQZSA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.441805237 (128h13m10.269513634s) [noise=true , address=45.91.116.8:4057 , piece_id=M6T6WCUNQCWAHD7EIBEPNZDROJYSKYNU6RMZEA57NF2MY3U4UQBA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.441830901 (128h13m10.269539297s) [noise=true , address=77.90.31.19:4855 , piece_id=PLE4DF4KOJ7PSL7BZZ27OPDEPAY5TS3MXVS6CJOYE7MKVI6EMKVQ, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.442204330 (128h13m10.269912723s) [noise=true , address=77.90.31.16:4749 , piece_id=7ZLE47BGQFUZPUWFSNPNLQATORK47S3SKAUBXE3UYDWTQZTFYQ4Q, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.442225574 (128h13m10.269933964s) [noise=true , address=45.91.116.13:4172 , piece_id=UAS6AYMPU63V74RBNESOVJI6OLFRGMLXFU4VODOXSLU4SJPFGP2Q, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.442570641 (128h13m10.270279045s) [noise=true , address=77.90.31.15:4717 , piece_id=SFN4GUUEXK3UWDD4UARHHRJ342O7HFETKZFY24NNMNMNFLKKUN6A, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.442921785 (128h13m10.270630186s) [noise=true , address=145.79.227.21:4492 , piece_id=I62BR2TA7DMKBIZ3I6PWRDXMKWESFLDIQOT6G5GYHIUV7D7RTPYA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.443312849 (128h13m10.271021245s) [noise=true , address=77.90.31.11:4578 , piece_id=YFTPF2BDAU5K5OUXPRW2CWICFQJDHZET26XSOBARGA4J7LEDUSEQ, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.443660144 (128h13m10.271368541s) [noise=true , address=145.79.227.30:4765 , piece_id=75YHVESRAD47ISWI4LBN5BC7MO4TOHN4MZIZH4XZHTHOFURTYERA, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.444132063 (128h13m10.271840464s) [noise=true , address=77.90.31.13:4629 , piece_id=2YTQW5ZPBHTUCKO3JJU7GKYFDZXHV3I3XEWYWVKTGN736IX6EEZQ, seg_pos={0 0}, upload=133328]: Uploading piece...
Dec 10 23:41:09.505011041 (128h13m10.332719421s) [upload=133328]: Next returned nil segment
Dec 10 23:41:09.505026797 (128h13m10.332735188s) [upload=133328]: Waiting for error group managing segments...
Dec 10 23:41:09.509423889 (128h13m10.33713228s) [noise=true , address=45.91.116.6:4984 , piece_id=PEPDXX3MKFARABFQKZ2AGWNA7LLX5W7DFEZQDA34B7FTC6TRSAEA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:09.525368549 (128h13m10.35307694s) [noise=true , address=145.79.227.4:4193 , piece_id=LNKONJYHPKJ2J2EMEDVV5WOETCFWMKQOKS4GPWIEMW3EYRDHYVZQ, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:09.525416761 (128h13m10.353125156s) [noise=true , address=145.79.227.21:4492 , piece_id=I62BR2TA7DMKBIZ3I6PWRDXMKWESFLDIQOT6G5GYHIUV7D7RTPYA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:09.526809497 (128h13m10.354517874s) [noise=true , address=45.91.116.8:4057 , piece_id=M6T6WCUNQCWAHD7EIBEPNZDROJYSKYNU6RMZEA57NF2MY3U4UQBA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:09.528356984 (128h13m10.356065375s) [noise=true , address=45.91.116.13:4172 , piece_id=UAS6AYMPU63V74RBNESOVJI6OLFRGMLXFU4VODOXSLU4SJPFGP2Q, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:09.596236324 (128h13m10.42394472s) [noise=true , address=5.231.73.5:4004 , piece_id=26RTHLRR5HPKIXCLIOKL4WXUK3J2A3EUE6A25FXCHOT3LPPTSSPA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:10.050191974 (128h13m10.877900365s) [noise=true , address=64.49.13.170:4725 , piece_id=AKA4YLYUG4NK2OC5PM3XWGMZI7VG34T6ZTAOELCYYEL6NQZYFXZA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:10.459713328 (128h13m11.28742172s) [noise=true , address=145.79.227.30:4765 , piece_id=75YHVESRAD47ISWI4LBN5BC7MO4TOHN4MZIZH4XZHTHOFURTYERA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:41:15.565240249 (128h13m16.392948648s) [noise=true , address=64.49.13.169:4709 , piece_id=VWM5U5CPKOYY3UDHLVKHL6V3ZCD5GXALKVPRBO37LUUZILGOSXZA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: <nil>
Dec 10 23:42:09.378807607 (128h14m10.206516007s) [noise=true , address=77.90.31.14:4694 , piece_id=OJBGBS6OLLZN5WTQR2WFLKYPKCA2J66NAHNGXFOTWRI5NLXAQRWA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:12qRMWzWyft2skqPXqUgPp41JY8faKf5K2En25vFdxp5dArE5SV, address:77.90.31.14:4694): protocol: write tcp 159.223.143.39:53338->77.90.31.14:4694: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.378870976 (128h14m10.206579368s) [noise=true , address=77.90.31.19:4855 , piece_id=PLE4DF4KOJ7PSL7BZZ27OPDEPAY5TS3MXVS6CJOYE7MKVI6EMKVQ, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 159.223.143.39:42682->77.90.31.19:4855: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.379371825 (128h14m10.207080222s) [noise=true , address=77.90.31.11:4578 , piece_id=YFTPF2BDAU5K5OUXPRW2CWICFQJDHZET26XSOBARGA4J7LEDUSEQ, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, address:77.90.31.11:4578): protocol: rpc: dial tcp 77.90.31.11:4578: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.379461630 (128h14m10.207170027s) [noise=true , address=77.90.31.16:4749 , piece_id=7ZLE47BGQFUZPUWFSNPNLQATORK47S3SKAUBXE3UYDWTQZTFYQ4Q, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:1EWSdFD1GDpqBHwXHZiDzR9nj8wE4vzmwnMbTrXrveiQXkUVC7, address:77.90.31.16:4749): protocol: rpc: dial tcp 77.90.31.16:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.379722368 (128h14m10.207430769s) [noise=true , address=77.90.31.18:4830 , piece_id=7ETI4WFQJM2FIYS2H4BLHA7AYDP7HHY3SGVUBRJXLBBDMJUQOY7Q, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, address:77.90.31.18:4830): protocol: write tcp 159.223.143.39:53002->77.90.31.18:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.379954616 (128h14m10.207663016s) [noise=true , address=77.90.31.21:4896 , piece_id=6QAN2QZQ2SZU4FUIZHBRCCVP4EUTKMK4GDILUJVHZCPRDKCBQZSA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE, address:77.90.31.21:4896): protocol: write tcp 159.223.143.39:44236->77.90.31.21:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.380115996 (128h14m10.207824396s) [noise=true , address=77.90.31.10:4530 , piece_id=WOWZQTYQRI3GH7JMGCJU43SNQYSKUJ5ZWLH3YUVDLM3C2OJMB7QA, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: write tcp 159.223.143.39:48214->77.90.31.10:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.380257939 (128h14m10.207966336s) [noise=true , address=77.90.31.15:4717 , piece_id=SFN4GUUEXK3UWDD4UARHHRJ342O7HFETKZFY24NNMNMNFLKKUN6A, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 159.223.143.39:43080->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 10 23:42:09.381071918 (128h14m10.208780319s) [noise=true , address=77.90.31.13:4629 , piece_id=2YTQW5ZPBHTUCKO3JJU7GKYFDZXHV3I3XEWYWVKTGN736IX6EEZQ, seg_pos={0 0}, upload=133328]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: context deadline exceeded; context deadline exceeded
Dec 10 23:42:09.381125376 (128h14m10.208833766s) [seg_pos={0 0}, upload=133328]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil>
Dec 10 23:42:09.401668908 (128h14m10.22937732s) [upload=133328]: Done uploading segments...
Dec 10 23:42:09.401726675 (128h14m10.229435072s) [upload=133328]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9)
Duration: 1m0.025927229s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................XXX......................] duration:
528.235842ms trace:
https://jaeger.dev.storj.io/trace/3e967eec712b2e24
FAIL (2025-12-10 23:42:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 528.263105ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................X....X...X........X...X...] duration:
490.825696ms trace:
https://jaeger.dev.storj.io/trace/4f84f1680308bcc4
FAIL (2025-12-10 23:41:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 490.848533ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................................XXX.X....................] duration:
564.268625ms trace:
https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8
FAIL (2025-12-10 23:41:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 564.293501ms
core
uplink
us-select-1
us1-placement-20
|