Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
1.704074321s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:29:04 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.704102363s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
1.543743865s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:28:47 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.543770339s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
37.183491ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:28:30 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 37.210706ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
1.353247496s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:28:28 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.353310171s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
30.257157ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:28:14 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 30.294016ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
1.319512034s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:27:49 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.319540295s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
1.609591847s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:27:43 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.609689717s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
1.312760411s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:27:12 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.312811007s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
32.782247ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:27:07 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 32.850043ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
1.349612963s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:26:52 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.349730077s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
35.292668ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:26:48 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.385156ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.39255853s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:26:10 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.392582577s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
49.2696475s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:26:08 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 49.26967871s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
35.962942ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:25:52 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 35.994241ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.760608397s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:25:47 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.760638717s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
1.323624971s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:24:45 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.323690183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.23657851s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:24:43 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1.23662529s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001003318s trace:
https://jaeger.dev.storj.io/trace/62357518aac8ca5f
FAIL (2026-03-25 14:24:42 UTC)
orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS======
Duration: 1m0.001089592s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
34.110101ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:24:38 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 34.168522ms
core
satellite
us1
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
28.001429ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:24:08 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 28.031185ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
1.329540168s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:23:34 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.329563391s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
41.45199ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:23:01 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 41.502373ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.398388614s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:22:48 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.398435257s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
39.017675ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:22:28 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 39.067327ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.415118484s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:21:53 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1.415144482s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.002637315s trace:
https://jaeger.dev.storj.io/trace/1d1f0bc4e4438495
FAIL (2026-03-25 14:21:52 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
upload=514680]: Uploading piece...
Mar 25 14:21:03.327216643 (459h21m0.923987655s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327580267 (459h21m0.924351301s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.327966258 (459h21m0.924737281s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328304150 (459h21m0.925075204s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328528343 (459h21m0.925299366s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328832656 (459h21m0.925603704s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.328990765 (459h21m0.925761788s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329286413 (459h21m0.926057443s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329346877 (459h21m0.9261179s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329706811 (459h21m0.926477834s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.329728635 (459h21m0.926499661s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330142941 (459h21m0.926913971s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330171985 (459h21m0.926943015s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330543280 (459h21m0.927314303s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330675401 (459h21m0.927446433s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.330936609 (459h21m0.927707633s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.331355022 (459h21m0.928126051s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Uploading piece...
Mar 25 14:21:03.402821184 (459h21m0.999592291s) [upload=514680]: Next returned nil segment
Mar 25 14:21:03.402848744 (459h21m0.99961978s) [upload=514680]: Waiting for error group managing segments...
Mar 25 14:21:03.408851473 (459h21m1.005622504s) [noise=true , address=77.90.31.16:4739 , piece_id=LEV3TZ2XFAYLQHULRJOG3DDBFMP3XRWX4EU2CQ2YOFICYXLIR3SA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.409008646 (459h21m1.005779679s) [noise=true , address=145.79.227.34:4916 , piece_id=645H2KOPV4YB2HTP3XZF2RF5ZICIR34QNYRNRKX5KHH2VANVVZOQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.418584053 (459h21m1.015355084s) [noise=true , address=77.90.31.17:4791 , piece_id=L5H6QECJGEQPTPRB6INQWHZPMK7ZXXDYYFFG5ANE37HAE33OTVXQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.446927214 (459h21m1.043698248s) [noise=true , address=45.91.116.7:4023 , piece_id=GYWVVXKLOAZSDJIS64INTFQIPQZCLBH7YPLTWY7QJSB766A3QTGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.448923805 (459h21m1.045694815s) [noise=true , address=77.90.31.14:4688 , piece_id=XDS6GE2YZK3LWUTBOMHLWTHM4ASCQJI25WTENUU75TJGFULYPOCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.458340727 (459h21m1.055111751s) [noise=true , address=45.91.116.12:4254 , piece_id=UIQ3FWZSZ52RJYWFVWNH3OGNQPZSDYEZEDX35K3R7SDKRIW3WOAQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.468833670 (459h21m1.065604702s) [noise=true , address=77.90.31.11:4567 , piece_id=ZSITPEN2OCYCHLSV3PDXCAXBAHEAQMS3YFCCOOE2SISJMUFRPHOA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.925971986 (459h21m1.522743012s) [noise=true , address=193.149.164.66:4984 , piece_id=CGM44RIHZGETSCKRZQOFL3VYIJTSB57RQDXTG3OPX3F3HJNQDYZA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.928271525 (459h21m1.525042552s) [noise=true , address=64.49.13.140:4509 , piece_id=7HN3FXTKC4RRUJZ34AHFDJHMUBIOWIQVFST626CR7X2ZYQS3CFCQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929087375 (459h21m1.525858388s) [noise=true , address=193.149.164.70:4120 , piece_id=HMFOJCRQTG3PAQDIZERNEYWKFFBUIRKRXP56OQUQWH6R2YX6NYLQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.929851130 (459h21m1.526622145s) [noise=true , address=193.149.164.71:4296 , piece_id=RXZXV3FCVDEFHBJQVGIWEKS57EJ3IKAJFTDGMOEZHRRBHV3NRDSA, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.935607616 (459h21m1.532378643s) [noise=true , address=193.149.164.69:4085 , piece_id=KGCVBEK344OW573PN2Y55JA7AC3ZTQD36G3T6TUHF7ZMZ4WN7N5A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.938874518 (459h21m1.535645546s) [noise=true , address=193.149.164.67:4030 , piece_id=PBIDHXYQ22HHK3HBJLFPC5X4L74Q332LNNEDYPFDV6SHWCQKRCNQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950243460 (459h21m1.547014493s) [noise=true , address=64.49.13.26:4955 , piece_id=VIN3RVGRSUUHYCHPXCDNVSPRLFSCH3JXIDERBV7NIM4K33ZUBUGQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: <nil>
Mar 25 14:21:03.950277580 (459h21m1.54704861s) [seg_pos={0 0}, upload=514680]: Satisfied cohort requirements after 14 pieces
Mar 25 14:21:03.950707500 (459h21m1.547478532s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WPfYrCJrhPTqZ2qgBvXFCUodKz6QeXCnkKgToFs5muNRSSLtK): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950721648 (459h21m1.547492674s) [noise=true , address=193.149.164.68:4050 , piece_id=F4AQMCLJL6ML2LQKWUIKJAN6C2Y3M7UVYXRCRVRON5BC6CLTEK6Q, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950793002 (459h21m1.547564034s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RcgfaeKP3XpZGuSB9GMURCCsjjc79WeZasUL2RRiQVDwxbnAk): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950804838 (459h21m1.547575864s) [noise=true , address=64.49.13.27:4964 , piece_id=UOBMO3UGR2SVYEXRSW2SQMHSDJV47OXOZRZJHCOHVWNXMIIDSA2A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.950899119 (459h21m1.547670144s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.950907089 (459h21m1.547678114s) [noise=true , address=145.79.227.44:4201 , piece_id=SHQIEKFZSOKO7SUBGDC7GSJQNL47IEWENP5KN5ZN5ZOS4K3R7Y7A, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951133312 (459h21m1.547904347s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12taemzrausuc58d8Dyo8xaR7MqDf72f37rEheF32sUfy8Z8GZP): protocol: expected piece hash; context canceled; EOF
Mar 25 14:21:03.951146101 (459h21m1.547917132s) [noise=true , address=64.49.13.138:4487 , piece_id=NMZRB7X7MX2KJRDEUBROG3F6ZPQOGYJLTJ5WHBBFOUQCKXIHIZKQ, seg_pos={0 0}, upload=514680]: Long tail cancellation due to optimal threshold
Mar 25 14:21:03.951175027 (459h21m1.547946039s) [seg_pos={0 0}, upload=514680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Mar 25 14:21:03.951282270 (459h21m1.548053293s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Mar 25 14:21:03.951290421 (459h21m1.548061457s) [upload=514680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Mar 25 14:21:04.013599996 (459h21m1.610371022s) [upload=514680]: Done uploading segments...
Mar 25 14:21:04.013621043 (459h21m1.610392066s) [upload=514680]: Upload finished. err: <nil>
Duration: 1m0.0026606s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
46.222749ms target:
https://us1.storj.io/login trace:
https://jaeger.dev.storj.io/trace/21096334d2fd8925
FAIL (2026-03-25 14:21:16 UTC)
orbiter warning: Get "https://us1.storj.io/login": dial tcp 34.4.32.59:443: i/o timeout
Duration: 46.276702ms
core
satellite
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
295.033637ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:08:24 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 295.140502ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
279.386445ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:07:26 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 279.455323ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
236.636868ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:06:49 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 236.691848ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
271.13154ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:06:05 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 271.171745ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
304.969363ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:05:30 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 305.002636ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
284.717778ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:04:18 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 284.749323ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
272.492571ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:03:09 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 272.524292ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
4.45980004s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 14:03:06 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.459833488s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
4.311869672s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 14:02:30 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.311903449s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
235.048956ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:02:09 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 235.120344ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
294.027705ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:01:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 294.068487ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.243073848s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 14:01:49 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.24311005s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
258.31298ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 14:00:58 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 258.343477ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
4.422598101s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 14:00:42 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.422633789s
core
uplink
ap1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
4.346880967s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 14:00:01 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.346917492s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
245.142907ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 13:59:45 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 245.256841ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
343.870641ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 13:59:27 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 343.899689ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
4.343276575s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 13:58:47 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.34337068s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
264.361438ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 13:58:30 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 264.415776ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
4.408345048s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 13:58:28 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.408373151s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
468.626531ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 13:57:25 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 468.806921ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
4.243689514s trace:
https://jaeger.dev.storj.io/trace/3241016a24cb2e0a
FAIL (2026-03-25 13:57:21 UTC)
orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded
Duration: 4.243723101s
core
uplink
ap1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
417.476772ms trace:
https://jaeger.dev.storj.io/trace/4b1ffae4fc329533
FAIL (2026-03-25 13:57:09 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context canceled ======LOGS====== Mar 25 13:20:57.428007150 (458h20m55.024778176s) [upload=513556]: Starting upload Mar 25 13:20:57.428130514 (458h20m55.024901544s) [upload=513556]: Done starting upload Mar 25 13:20:57.428299046 (458h20m55.025070066s) [upload=513556]: Uploading segments... Mar 25 13:20:57.428378156 (458h20m55.025149181s) [upload=513556]: Got next segment. Inline: true Mar 25 13:20:57.428391374 (458h20m55.025162399s) [upload=513556]: Waiting for error group managing segments... Mar 25 13:20:57.428497332 (458h20m55.025268363s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Mar 25 13:20:57.428506270 (458h20m55.025277301s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Mar 25 13:20:57.428511663 (458h20m55.025282691s) [upload=513556]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Mar 25 13:20:57.468762737 (458h20m55.065533769s) [upload=513556]: Done uploading segments... Mar 25 13:20:57.468831633 (458h20m55.065602664s) [upload=513556]: Upload finished. err: metaclient: context canceled
Duration: 417.493954ms
core
uplink
us-select-1
us1-placement-20
|