Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.X............................] duration:
375.916058ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:55:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 375.947114ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................X...XX.X........................] duration:
445.935586ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:55:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 445.96611ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
515.951393ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:55:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 515.976395ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [............................X..X............................] duration:
521.666027ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:55:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 521.688177ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X.....X...............................] duration:
441.924333ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:55:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 441.998276ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................X...XX.X.......................] duration:
397.895539ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:54:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 397.920526ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................XX..X.X..............................] duration:
443.593706ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:54:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 443.616561ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X.....X..............................] duration:
457.758476ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:54:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 457.786249ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.............................X..X...........................] duration:
510.121638ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:54:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.142729ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X.....X.............................] duration:
471.850012ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:54:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 471.876052ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration:
417.396929ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:54:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 417.426468ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................XX..X.X.............................] duration:
534.898384ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:54:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 534.924369ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..............................X..X..........................] duration:
510.786751ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:54:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 510.81444ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
341.268135ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:53:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 341.354858ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............................X...XX.X......................] duration:
410.793168ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:53:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 410.815339ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X.....X............................] duration:
363.387061ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:53:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 363.41045ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................XX..X.X............................] duration:
429.694738ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:53:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 429.716136ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration:
366.460006ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:53:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 366.482225ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................X..X.........................] duration:
561.820239ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:53:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 561.84027ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
349.62411ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:52:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 349.640473ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X.....X...........................] duration:
459.396655ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:52:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 459.427368ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X.X.........................] duration:
344.412178ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:52:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 344.43817ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................X...XX.X.....................] duration:
410.236045ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:52:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 410.261664ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X.....X..........................] duration:
351.55703ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:52:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 351.712896ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................XX..X.X...........................] duration:
436.01923ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:52:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 436.045051ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
481.792384ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:52:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 481.826724ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X..X........................] duration:
548.684755ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:52:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 548.713706ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X.X........................] duration:
459.13976ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:52:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 459.173211ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [................................X...XX.X....................] duration:
398.269277ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:51:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 398.297139ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................XX..X.X..........................] duration:
396.992727ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:51:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 397.019331ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
342.928077ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:51:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 342.953517ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X.....X.........................] duration:
443.583282ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:51:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 443.613186ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.................................X..X.......................] duration:
502.948531ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:51:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 502.973759ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration:
407.833451ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:50:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 407.854055ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.................................X...XX.X...................] duration:
410.374329ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:50:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 410.402426ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X.....X........................] duration:
398.652927ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:50:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 398.691269ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................XX..X.X.........................] duration:
436.444519ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:50:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 436.473934ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.X......................] duration:
399.96858ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:50:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 399.994563ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................................X...XX.X..................] duration:
404.198372ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:50:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 404.290451ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [..................................X..X......................] duration:
523.69413ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:50:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 523.715763ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
449.026667ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:50:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 449.056181ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration:
347.682311ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:49:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 347.785691ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
480.014869ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:49:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 480.153506ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X.....X.......................] duration:
380.896152ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:49:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 380.922298ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................................X...XX.X.................] duration:
408.972206ms trace:
https://jaeger.dev.storj.io/trace/67546c94dad79c68
FAIL (2025-11-22 12:49:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 408.999162ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.....X......................] duration:
357.393234ms trace:
https://jaeger.dev.storj.io/trace/3b4bbba8233c3252
FAIL (2025-11-22 12:49:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 4)
======LOGS======
5231 (266h46m18.327848666s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560386882 (266h46m18.328070315s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560643615 (266h46m18.328327043s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.560964939 (266h46m18.328648377s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561295970 (266h46m18.328979421s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561543171 (266h46m18.329226606s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.561992295 (266h46m18.329675715s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562090500 (266h46m18.329773935s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.562638716 (266h46m18.330322151s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.563244074 (266h46m18.330927509s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.564399751 (266h46m18.332083176s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Uploading piece...
Nov 22 12:32:53.636343111 (266h46m18.40402654s) [upload=277533]: Next returned nil segment
Nov 22 12:32:53.636356267 (266h46m18.404039692s) [upload=277533]: Waiting for error group managing segments...
Nov 22 12:32:53.643835394 (266h46m18.41151882s) [noise=true , address=145.79.227.19:4099 , piece_id=KFB7EKK7IRGQIRRFVIFFIFKSK2EAXO35EMOPKHGYIYVCCZK2Z7AA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:53.644997608 (266h46m18.412681034s) [noise=true , address=145.79.227.15:4887 , piece_id=7JWASURMCGJB7FUZGD7OF2Z5TNWTCJDJJDMYIW2MZ6XJJE552DRQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218186589 (266h46m18.985870034s) [noise=true , address=64.49.13.170:4724 , piece_id=NYDIBEUOONSZIEVX2WBCUXQZN43M6ZTCLPYB4VCXJKKCXFE44ZNA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:32:54.218512631 (266h46m18.98619606s) [noise=true , address=64.49.13.27:4967 , piece_id=LL5SPMIUILPYXM6J2FBOLJ7UKKEHROYLPG5UOFEFOTPU4NSFZQIA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: <nil>
Nov 22 12:33:53.499194349 (266h47m18.266877779s) [noise=true , address=77.90.31.21:4889 , piece_id=JHYHKDPHC5VPEHYNZVBGCT42DD2J2ZOMHNYFBJ77PNVRMVF2U36Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1wcX8iEhi5LrftcXUQE3Lrm4igNwhDtvT3RNBERTGsKyQ6KLj6, address:77.90.31.21:4889): protocol: rpc: dial tcp 77.90.31.21:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499375551 (266h47m18.267058979s) [noise=true , address=77.90.31.19:4856 , piece_id=OYQOBBFXFUIAB5C7REVJOJA674725OAPJRWN46OULFC5KI25NFDA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: rpc: dial tcp 77.90.31.19:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499556037 (266h47m18.267239473s) [noise=true , address=77.90.31.11:4582 , piece_id=ALT4PKDZQCAUZTAKIW2F5HRAM4YR6U5OG5J4JK3ARXTPNHPUDKBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, address:77.90.31.11:4582): protocol: rpc: dial tcp 77.90.31.11:4582: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499581094 (266h47m18.267264525s) [noise=true , address=45.91.116.8:4051 , piece_id=4TQKZVU2NEQS3XKNJBR4Y66YWUHUFVTPLNW4CODY7SIRZ3XU6FOA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3, address:45.91.116.8:4051): protocol: rpc: dial tcp 45.91.116.8:4051: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499743731 (266h47m18.267427159s) [noise=true , address=45.91.116.14:4217 , piece_id=5G2P33SPGTDMIATW2IN4UG5USENAP6NEGNKJCOLP5GFQKF4VZK4A, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE, address:45.91.116.14:4217): protocol: rpc: dial tcp 45.91.116.14:4217: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.499881469 (266h47m18.267564894s) [noise=true , address=45.91.116.12:4252 , piece_id=4LSOU54RSM5KLPXJIXJLU26QAUYHCKTUGIOVI4QJ6RUJ4B5POOGQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1XYnnwB1yzhc6p4eFg5tkxLFd2Wrt5Mtwwd2BjyHmJgro6q8kF, address:45.91.116.12:4252): protocol: write tcp 159.223.143.39:46050->45.91.116.12:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500076781 (266h47m18.267760206s) [noise=true , address=45.91.116.7:4041 , piece_id=MH2OIHWTUK32IY23VFIDC7VD7N7OYF3FH2DXS67OT5FGIUHHB2OQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1BvWxeWEnE4X8CJA6KJKryewizvzdGnbmthSZmr2Rfb7LdtUHV, address:45.91.116.7:4041): protocol: rpc: dial tcp 45.91.116.7:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500221536 (266h47m18.267904968s) [noise=true , address=45.91.116.5:4307 , piece_id=BHSQREJZCKSDE7XLOA7GH3KEF344LTIP3NPJV5GJONJQ5BPFIH3Q, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: rpc: dial tcp 45.91.116.5:4307: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500437209 (266h47m18.268120637s) [noise=true , address=77.90.31.13:4638 , piece_id=JJBQGDNPCKBN3VJODPNMOSO34NPM24EY5EWL5RSIXY2VXXAEXETA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:14HQS4QNDPnS5W7Ae2XURGNModFwarBsbycx7GUFRmkjvKYdKF, address:77.90.31.13:4638): protocol: rpc: dial tcp 77.90.31.13:4638: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500592956 (266h47m18.268276393s) [noise=true , address=45.91.116.6:4985 , piece_id=MT6SE3XLBQOSRIXIESJT3IIKPDXEAM5LG7JEQTGRDIPA2ABPTARA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp, address:45.91.116.6:4985): protocol: rpc: dial tcp 45.91.116.6:4985: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500603529 (266h47m18.268286961s) [noise=true , address=77.90.31.17:4777 , piece_id=Y2LJF75NHJLPDCBP546RJBDUYKXUWZ4GTBIC4VKK6QDV6IM4FYPQ, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: rpc: dial tcp 77.90.31.17:4777: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500759304 (266h47m18.268442732s) [noise=true , address=45.91.116.11:4142 , piece_id=HLPVKTMZ2CNUK5SEXDLSEXMVZ7EKYJDOCEF57KHQIEX73K4OHBUA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:135mYyE4Gy1gcCXSocnb41e9u4z4MpqMjodmaRnrNs9AGS1YcX, address:45.91.116.11:4142): protocol: rpc: dial tcp 45.91.116.11:4142: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500911284 (266h47m18.268594712s) [noise=true , address=77.90.31.14:4688 , piece_id=I4LNKZXPDASXCKGAHBGFPH67RJZMQYXTATK2GLNGRE5DMLQZHGTA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, address:77.90.31.14:4688): protocol: rpc: dial tcp 77.90.31.14:4688: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.500999071 (266h47m18.268682499s) [noise=true , address=77.90.31.10:4549 , piece_id=OJHZQCIRQRRUWCZHJQ7XUS7JO76LZH23ZSLGVYXIHIETFZQH5OBA, seg_pos={0 0}, upload=277533]: Done uploading piece. err: ecclient: upload failed (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy, address:77.90.31.10:4549): protocol: write tcp 159.223.143.39:41216->77.90.31.10:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:53.501028403 (266h47m18.268711821s) [seg_pos={0 0}, upload=277533]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil>
Nov 22 12:33:53.522282740 (266h47m18.289966165s) [upload=277533]: Done uploading segments...
Nov 22 12:33:53.522325473 (266h47m18.290008901s) [upload=277533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4)
Duration: 357.4263ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................XX..X.X........................] duration:
471.001076ms trace:
https://jaeger.dev.storj.io/trace/560a431e0a1550ec
FAIL (2025-11-22 12:49:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 471.028347ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration:
524.50924ms trace:
https://jaeger.dev.storj.io/trace/49794518b66817f9
FAIL (2025-11-22 12:49:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 524.540192ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
339.653024ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:48:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 339.683478ms
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration:
388.359446ms trace:
https://jaeger.dev.storj.io/trace/66d9faa943161cb
FAIL (2025-11-22 12:48:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 7)
======LOGS======
Nov 22 12:33:01.330811854 (266h46m26.09849529s) [upload=277537]: Starting upload
Nov 22 12:33:01.330902729 (266h46m26.09858616s) [upload=277537]: Done starting upload
Nov 22 12:33:01.331240897 (266h46m26.098924325s) [upload=277537]: Uploading segments...
Nov 22 12:33:01.331641290 (266h46m26.099324717s) [upload=277537]: Got next segment. Inline: false
Nov 22 12:33:01.331736518 (266h46m26.099419992s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:33:01.331743323 (266h46m26.099426736s) [upload=277537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:33:01.357467557 (266h46m26.125150991s) [seg_pos={0 0}, upload=277537]: Begin upload segment...
Nov 22 12:33:01.357562052 (266h46m26.125245488s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.358813185 (266h46m26.126496611s) [seg_pos={0 0}, upload=277537]: Done begin upload segment.
Nov 22 12:33:01.358887112 (266h46m26.126570543s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359076930 (266h46m26.126760372s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359460721 (266h46m26.127144156s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359602229 (266h46m26.127285663s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.359865179 (266h46m26.127548608s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360067936 (266h46m26.127751371s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360335100 (266h46m26.128018531s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360488734 (266h46m26.128172165s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360769262 (266h46m26.12845269s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.360930558 (266h46m26.12861399s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361101285 (266h46m26.128784714s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361371227 (266h46m26.129054653s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361478968 (266h46m26.129162397s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361857376 (266h46m26.129540809s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.361992638 (266h46m26.129676075s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362287715 (266h46m26.129971148s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.362381270 (266h46m26.130064698s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Uploading piece...
Nov 22 12:33:01.444134822 (266h46m26.211818252s) [upload=277537]: Next returned nil segment
Nov 22 12:33:01.444165621 (266h46m26.211849054s) [upload=277537]: Waiting for error group managing segments...
Nov 22 12:33:01.452636815 (266h46m26.220320241s) [noise=true , address=145.79.227.49:4354 , piece_id=C5GBHZPLBPHJUFC5RYMTKNIAB56SVZDDEJXJQRQYJL44OFRTDRNA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453306000 (266h46m26.220989428s) [noise=true , address=145.79.227.19:4083 , piece_id=Q7PYXDDPR43MXK6UEHZTKICNGLTQ2XH5CHYJGGRH4TUWUL2OZ7OQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453375624 (266h46m26.221059036s) [noise=true , address=145.79.227.2:4133 , piece_id=EMRVT4OQYYHZHNHPSBRKAJ33Q4PSFB2XNYUGA3BWTPLEFLNVXO7A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.453474145 (266h46m26.221157602s) [noise=true , address=145.79.227.14:4856 , piece_id=BHIKFKNQCYPQ6KPXRN7TIORPSJ44CXPQNS4DNMTOLKMKEPQ32HAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:01.454204627 (266h46m26.221888059s) [noise=true , address=145.79.227.3:4167 , piece_id=F3AQWB2TPA7XKSEHOY7KIRXU3AHNENYGL237LGLQQ25JBNM6QAAQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:54.943657304 (266h47m19.711340729s) [noise=true , address=45.91.116.6:4986 , piece_id=LULETP6O53HIIWU4UGZKANHEIH3KQQZK5NCM7EJJF2AVBXUTUETQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:33:55.138654837 (266h47m19.906338262s) [noise=true , address=77.90.31.16:4744 , piece_id=Y2T7BTX3NPMN4IXS57N36Y6U3VH3MXPD6QDZNUQLE2DDKJGVNRMQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: <nil>
Nov 22 12:34:01.299549282 (266h47m26.067232716s) [noise=true , address=77.90.31.12:4624 , piece_id=BHV5IITYUQY6QZ7NT5VMOZSKCYA3H3IPJSZVQQE3JX5SUZXSQ6GQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299764337 (266h47m26.06744777s) [noise=true , address=45.91.116.9:4274 , piece_id=6MZWDUNSXH5TWKVHEAM42ROYWXY6RBJAMQOKXFQE6ADINU5PK7SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12g6jFa38dTz5UdgsQaEuM2MSH1H7cxaysWsVmbHkM2Dd2W2iDb, address:45.91.116.9:4274): protocol: rpc: dial tcp 45.91.116.9:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.299879271 (266h47m26.067562709s) [noise=true , address=77.90.31.14:4685 , piece_id=CZLTN6M2R6BN543NDTESOYMLFYTUPCZ7CAVEUDN5WIWMHOCZU2BA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300076046 (266h47m26.067759477s) [noise=true , address=77.90.31.18:4824 , piece_id=EREFJH3W22G4DF6Y3PZWMBBKXMYYEK62XDMSYP2M6LLPGA76UB2A, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300228343 (266h47m26.067911791s) [noise=true , address=77.90.31.15:4703 , piece_id=YJKOM226WH5E4EYSQBD5XNXAB5MZEQCE36E3PX4BL67OI3NFXTYA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, address:77.90.31.15:4703): protocol: rpc: dial tcp 77.90.31.15:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300302645 (266h47m26.067986077s) [noise=true , address=77.90.31.13:4644 , piece_id=OJQBNYOCFT3RSQWAWVSTA4KPQVU7SP44ZQAUVB46EY4O7PK2SSZA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:13QCH91kRMQjVj2ASsVDcyXQ1aapQRVeUm6MMCGimsR6Ns973h, address:77.90.31.13:4644): protocol: write tcp 159.223.143.39:57130->77.90.31.13:4644: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300508422 (266h47m26.068191856s) [noise=true , address=77.90.31.10:4536 , piece_id=VRGUWVT6T73VNEF2GMWVI7CO42IVJUB3VCSC7GK2NX4YPHV7YQDA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: rpc: dial tcp 77.90.31.10:4536: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300525931 (266h47m26.068209365s) [noise=true , address=77.90.31.21:4909 , piece_id=OBAFRRD5L2S3W7XVEU73HYJWOAYS7YQSRUWJ5L6HKMHC77R7GVCQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:36518->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.300748211 (266h47m26.068431645s) [noise=true , address=77.90.31.11:4560 , piece_id=7MKKT426DRM6EEUQNUUGCFXUNVZGKPKRB5COEWD7HHBFXMWQ5EUA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, address:77.90.31.11:4560): protocol: rpc: dial tcp 77.90.31.11:4560: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301048821 (266h47m26.068732252s) [noise=true , address=45.91.116.13:4157 , piece_id=V3ZTAHZY3HBBQ2X77LMX64EJ6GPPWWH2JYDBUEO5YY6IQ7JQL3XQ, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1ZMuuRZWAmRdQBHg2PFQCq9gB1uwgDTnZ72bm2ghMv1LRkewUj, address:45.91.116.13:4157): protocol: rpc: dial tcp 45.91.116.13:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301216258 (266h47m26.068899686s) [noise=true , address=45.91.116.10:4091 , piece_id=ONXHFIFUUXNMFVNJVN2HT3EYSYBLZIM6JFHG6IFUDG3ENZG3B2SA, seg_pos={0 0}, upload=277537]: Done uploading piece. err: ecclient: upload failed (node:1gPa2A8V3hY6WFgDTMk5ZMQpEaVVuhjXFQYHoZhK2R9JUPyepz, address:45.91.116.10:4091): protocol: rpc: dial tcp 45.91.116.10:4091: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:34:01.301245005 (266h47m26.068928424s) [seg_pos={0 0}, upload=277537]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil>
Nov 22 12:34:01.321634820 (266h47m26.089318245s) [upload=277537]: Done uploading segments...
Nov 22 12:34:01.321685835 (266h47m26.08936926s) [upload=277537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7)
Duration: 388.396654ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|